You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Adam <ad...@gmail.com> on 2013/10/28 22:59:29 UTC
How to recover entire self contained CloudStack after major network changes?
Hi All,
I know what I'm about to ask may seem like crazy talk, but I have a
situation at work where our CloudStack "box" was moved to another network.
This single server is a self-contained cloud with everything on one box.
I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for primary and
secondary storage mounts. It's a very basic configuration, but my company
moved from one town to another, and the IT dept decided to move our server
to a different switch and subnet without asking. So now literally every
single IP address of any importance has been changed.
I've been trying in vain to get this CS up and running again, but it seems
like everything has changed including the guids, or at least that seems to
be the case since I cannot match up guids from the database to guids on the
primary/secondary NFS.
I'm sure if each piece were separate and moved one piece at a time, it
would be fairly straightforward, but since this box is self-contained and
everything changed at the same time, it is proving to be very difficult.
Does anyone have any suggestions for me to attempt to recover my CS?
Best Regards,
Adam Scarcella
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Adam <ad...@gmail.com>.
I actually did upgrade from 4.1.1 to 4.2, but had to roll back.
http://comments.gmane.org/gmane.comp.apache.cloudstack.user/7387
After the roll back, that's when my company moved to a new office and
changed networks. In the middle of me trying to get 4.1.1 back up again
after the move, I did upgrade to 4.2 (again) to see if I would have more
luck this time. So I am actually running on 4.2 now. I still cannot deploy
my system VMs however, despite following these instructions:
http://cloud.kelceydamage.com/cloudfire/blog/2013/10/08/conquering-the-cloudstack-4-2-dragon-kvm/
I actually performed that procedure several times thinking that perhaps I
missed something. The very first time I ran it, I could not umount /mnt
because I'm self contained, and my /export/primary storage is mounted there
as well. So... I realized that those mounting/umounting steps are merely to
gain access to the /export/secondary/Template/tmpl/1/3/ files, which are
local to my box. So the second time I performed the procedure, I simply
renamed the *.qcow2 file to match what was in my database.
mv 83f8aec6-9446-4fe5-bc9e-258d25c9348a.qcow2
5c331a9a-8d48-4e53-b9fa-b5e7655b9175.qcow2
Restarted my entire server (probably overkill, but I'm desperate) but that
still didn't fix it. I still cannot start system VMs.
I created a clean management-server.log file and found that it now thinks I
have 3 orphaned management hosts.
2013-10-29 14:48:41,679 DEBUG [utils.db.ConnectionConcierge]
(Cluster-Heartbeat-1:null) Registering a database connection for
ClusterManagerHeartBeat2
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Add orphan management server msid found in host
table to initial clustering notification, orphan msid: 279278805450800
2013-10-29 14:48:41,691 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_f278348c
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Add orphan management server msid found in host
table to initial clustering notification, orphan msid: 279278805451095
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Add orphan management server msid found in host
table to initial clustering notification, orphan msid: 279278805451190
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Found 3 inactive management server node based on
timestamp
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) management server node msid: 279278805450800,
name: null, service ip: orphan, version: null
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) management server node msid: 279278805451095,
name: null, service ip: orphan, version: null
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) management server node msid: 279278805451190,
name: null, service ip: orphan, version: null
2013-10-29 14:48:41,691 INFO [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Trying to connect to orphan
2013-10-29 14:48:41,693 ERROR [cloud.cluster.ClusterManagerImpl]
(Cluster-Heartbeat-1:null) Unexpected exception in cluster heartbeat
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at
com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1405)
at
com.cloud.cluster.ClusterManagerImpl.initPeerScan(ClusterManagerImpl.java:1036)
at
com.cloud.cluster.ClusterManagerImpl.access$1100(ClusterManagerImpl.java:95)
at com.cloud.cluster.ClusterManagerImpl$4.run(ClusterManagerImpl.java:786)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2013-10-29 14:48:41,699 DEBUG [utils.db.ConnectionConcierge]
(Cluster-Heartbeat-1:null) Registering a database connection for
ClusterManagerHeartBeat2
I kept looking through the logs and found this error repeated several times:
2013-10-29 14:49:40,072 INFO [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Insufficient capacity
com.cloud.exception.InsufficientAddressCapacityException: Insufficient
address capacityScope=interface com.cloud.dc.Pod; id=1
at
com.cloud.network.NetworkManagerImpl.fetchNewPublicIp(NetworkManagerImpl.java:472)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.assignPublicIpAddress(NetworkManagerImpl.java:376)
at
com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:186)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:153)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2157)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:887)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
I've checked my Zone, Pod, Cluster, Network settings, CIDRs, Management IP
Range, Guest IP Range, etc... a ton of times now, and cannot find out where
the problem may be.
Since the problem began when we moved the self-contained server from one
network to another, and now the errors suggest that I do not have enough
IPs (and I'm assuming it means I do not have enough public IPs based on the
stack trace "at
com.cloud.network.NetworkManagerImpl.fetchNewPublicIp(NetworkManagerImpl.java:472)"),
what table(s) do I need to look at and potentially clear out to fix this
issue?
-Adam
Best Regards,
Adam Scarcella
On Tue, Oct 29, 2013 at 11:59 AM, Andrija Panic <an...@gmail.com>wrote:
> my bad for secondary storage - there is no maintance mode for that :)
>
> I was just refering to line:
>
> host=10.97.38.10 - which is hopefully new IP address :)
>
> Since you got Server capactity issues, I suggest folowing, as temp
> solution:
>
> Check CPU alocation, RAM alocation, and increase overprovisioning factor at
> POD level (at least this is in 4.2, or only in global config as was in
> 4.0.0), so if you are running out of CPU/RAM resopurces, you might be able
> to fix this this way.
>
> I had this exact type of problem (SSVM not starting), when upgraded from
> 4,0 to 4.2,because of bad SSVM template - I had to download
>
> http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2(previously
> I had very old, which came with Cs 4.0.0) - but if you did not
> upgrade your CS, then I suppose this is not the case...
>
> Best
>
>
>
>
>
> On 29 October 2013 16:42, Adam <ad...@gmail.com> wrote:
>
> > Hi Andrija,
> >
> > Thanks for your reply.
> >
> > Here is the output of my agent.properties file (which looks correct to
> me):
> >
> > [root@cs-east-prod1 ~]# cat /etc/cloudstack/agent/agent.properties
> > #Storage
> > #Tue Oct 29 10:15:49 GMT-05:00 2013
> > guest.network.device=cloudbr0
> > workers=5
> > private.network.device=cloudbr0
> > port=8250
> > resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
> > pod=1
> > zone=1
> > guid=db24e08f-ac6e-3974-b223-274d49af52d0
> > cluster=1
> > public.network.device=cloudbr0
> > local.storage.uuid=296f0404-9658-4269-bc24-64b1247d5e3f
> > domr.scripts.dir=scripts/network/domr/kvm
> > host=10.97.38.10
> > LibvirtComputingResource.id=10
> >
> > [root@cs-east-prod1 ~]# ifconfig cloudbr0
> > cloudbr0 Link encap:Ethernet HWaddr AC:16:2D:84:D4:24
> > inet addr:10.97.38.10 Bcast:10.97.38.255 Mask:255.255.255.0
> > inet6 addr: fe80::ae16:2dff:fe84:d424/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:10216 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:10343 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:1605339 (1.5 MiB) TX bytes:24242332 (23.1 MiB)
> >
> >
> > I have 4.5 TB free on my 5.7 TB RAID so I haven't even come close to the
> > "pool.storage.allocated.capacity.disablethreshold" or
> > "pool.storage.capacity.disablethreshold" of "0.85"
> >
> > [root@cs-east-prod1 ~]# df -kh
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/mapper/vg_cloudstackuse-lv_root
> > 5.7T 921G 4.5T 17% /
> > tmpfs 64G 72K 64G 1% /dev/shm
> > /dev/sda1 485M 118M 342M 26% /boot
> > 10.97.38.10:/export/primary
> > 5.7T 921G 4.5T 17%
> > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> > [root@cs-east-prod1 ~]# cd /export/
> > [root@cs-east-prod1 export]# ls -l
> > total 8
> > drwxrwxrwx. 3 root root 4096 Oct 27 19:18 primary
> > drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondary
> > [root@cs-east-prod1 export]# du -sh *
> > 238G primary
> > 110G secondary
> > [root@cs-east-prod1 export]#
> >
> > I've manually destroyed the SSVM several times and yes CS does attempt to
> > create a new one, but the same issue occurs. Regardless of whether I
> > destroy the SSVM or if CS does it, it just keeps looping.
> >
> > I'm not exactly sure how to check if my secondary storage is in
> Maintenance
> > Mode. I know I can enable Maintenance Mode of my Primary Storage, but how
> > to check for the Secondary Storage?
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Tue, Oct 29, 2013 at 11:16 AM, Andrija Panic <andrija.panic@gmail.com
> > >wrote:
> >
> > > I suppose, you can recheck the network, and also check the AGENT
> > > configuration file, make changes
> > > there: /etc/cloudstack/agent/agent.properties
> > >
> > > From logs:
> > >
> > > InsufficientServerCapacityException: Unable to create
> > > > a deployment for VM.....
> > >
> > > I had this kind of error, when my storage pools were disabled once, and
> > > also when I have reached "disable capacity treshold" - my storage pool
> > was
> > > over 85% full.
> > >
> > > So check if you secondaru storge is in maintance mode, or similar...
> > >
> > > Also, destroying SSVM will force CS into making new one...
> > >
> > >
> > > On 29 October 2013 16:09, Adam <ad...@gmail.com> wrote:
> > >
> > > > Hi Geoff,
> > > >
> > > > Thanks for the reply, I was able to manually mount the secondary
> > storage
> > > > NFS:
> > > >
> > > > [root@cs-east-prod1 secondaryStorage]# hostname
> > > > cs-east-prod1
> > > > [root@cs-east-prod1 secondaryStorage]# hostname --fqdn
> > > > cs-east-prod1.na.tibco.com
> > > > [root@cs-east-prod1 secondaryStorage]#
> > > > [root@cs-east-prod1 ~]# mkdir /secondaryStorage
> > > > [root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
> > > > /secondaryStorage
> > > > [root@cs-east-prod1 ~]# cd /
> > > > [root@cs-east-prod1 /]# ls -l
> > > > total 120
> > > > dr-xr-xr-x. 2 root root 4096 Oct 23 04:48 bin
> > > > dr-xr-xr-x. 5 root root 3072 Oct 22 21:10 boot
> > > > drwxr-xr-x. 10 root root 4096 Jun 25 10:23 cgroup
> > > > drwxr-xr-x. 19 root root 3960 Oct 29 10:38 dev
> > > > drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
> > > > drwxrwxrwx. 4 root root 4096 Jul 3 20:10 export
> > > > drwxr-xr-x. 3 root root 4096 Jul 3 19:14 home
> > > > lrwxrwxrwx. 1 root root 25 Aug 28 15:46 Installers ->
> > > > /var/www/html/Installers/
> > > > lrwxrwxrwx. 1 root root 19 Jul 5 20:58 ISOs ->
> > /var/www/html/ISOs/
> > > > dr-xr-xr-x. 11 root root 4096 Jul 3 19:02 lib
> > > > dr-xr-xr-x. 10 root root 12288 Oct 28 03:33 lib64
> > > > drwx------. 2 root root 16384 Jul 3 18:44 lost+found
> > > > drwxr-xr-x. 3 root root 4096 Oct 27 12:05 media
> > > > drwxr-xr-x. 2 root root 0 Oct 29 10:08 misc
> > > > drwxr-xr-x. 39 root root 4096 Sep 6 12:07 mnt
> > > > drwxr-xr-x. 2 root root 0 Oct 29 10:08 net
> > > > drwxr-xr-x. 3 root root 4096 Jul 3 19:05 opt
> > > > drwxr-xr-x. 2 root root 4096 Oct 27 10:06 primarymount
> > > > dr-xr-xr-x. 557 root root 0 Oct 29 10:08 proc
> > > > dr-xr-x---. 35 root root 4096 Oct 29 10:09 root
> > > > dr-xr-xr-x. 2 root root 12288 Oct 28 03:33 sbin
> > > > drwxr-xr-x. 2 root root 4096 Oct 27 10:07 secondarymount
> > > > drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondaryStorage
> > > > drwxr-xr-x. 7 root root 0 Oct 29 10:08 selinux
> > > > drwxr-xr-x. 2 root root 4096 Sep 23 2011 srv
> > > > drwxr-xr-x. 13 root root 0 Oct 29 10:08 sys
> > > > drwxrwxrwt. 16 root root 4096 Oct 29 11:07 tmp
> > > > drwxr-xr-x. 13 root root 4096 Jul 3 18:55 usr
> > > > drwxr-xr-x. 23 root root 4096 Jul 3 19:45 var
> > > > [root@cs-east-prod1 /]# cd secondaryStorage/
> > > > [root@cs-east-prod1 secondaryStorage]# ls -l
> > > > total 12
> > > > drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
> > > > drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
> > > > drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
> > > > [root@cs-east-prod1 secondaryStorage]#
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
> > > > geoff.higginbottom@shapeblue.com> wrote:
> > > >
> > > > > Adam,
> > > > >
> > > > > System VMs looping and failing to create is often caused by the
> Host
> > > > being
> > > > > unable to access Sec Storage.
> > > > >
> > > > > Do a manual mount of the Sec Storage NFS on the Host to confirm it
> > can
> > > > > access it.
> > > > >
> > > > > Regards
> > > > >
> > > > > Geoff Higginbottom
> > > > >
> > > > > D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
> > > > >
> > > > > geoff.higginbottom@shapeblue.com
> > > > >
> > > > > -----Original Message-----
> > > > > From: Adam [mailto:adam.scarcella@gmail.com]
> > > > > Sent: 29 October 2013 14:44
> > > > > To: users@cloudstack.apache.org
> > > > > Subject: Re: How to recover entire self contained CloudStack after
> > > major
> > > > > network changes?
> > > > >
> > > > > I exported the db as a *.sql file, did search/replace and that
> seemed
> > > to
> > > > > help a lot, but still not 100% out of the woods. Now I'm hitting
> the
> > > > issue
> > > > > where the System VMs (SecondaryStorageVm & ConsoleProxy) do not
> > start.
> > > > They
> > > > > just keep looping from creation to being expunged.
> > > > >
> > > > > My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> > > > > Primary Storage is online:
> > > > >
> > > > > [root@cs-east-prod1 ~]# df -kh
> > > > > Filesystem Size Used Avail Use% Mounted on
> > > > > /dev/mapper/vg_cloudstackuse-lv_root
> > > > > 5.7T 921G 4.5T 17% /
> > > > > tmpfs 64G 72K 64G 1% /dev/shm
> > > > > /dev/sda1 485M 118M 342M 26% /boot
> > > > > 10.97.38.10:/export/primary
> > > > > 5.7T 921G 4.5T 17%
> > > > > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> > > > >
> > > > > Here is an excerpt from my most recent log:
> > > > >
> > > > > 2013-10-29 09:22:23,389 DEBUG
> > > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > > 2013-10-29 09:22:23,391 DEBUG
> > > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > > 2013-10-29 09:22:23,441 DEBUG
> > > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > > 2013-10-29 09:22:23,443 DEBUG
> > > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > > 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> > > > > (AgentManager-Handler-5:null) Ping from 10
> > > > > 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> > > > > (StatsCollector-2:null) VmStatsCollector is running...
> > > > > 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> > > > > (StatsCollector-3:null) StorageCollector is running...
> > > > > 2013-10-29 09:22:38,716 INFO
> > > [storage.endpoint.DefaultEndPointSelector]
> > > > > (StatsCollector-3:null) No running ssvm is found, so command will
> be
> > > sent
> > > > > to LocalHostEndPoint
> > > > > 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> > > > > (StatsCollector-1:null) HostStatsCollector is running...
> > > > > 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> > > > > (StatsCollector-1:null) Seq 10-768344083: Received: { Ans: ,
> MgmtId:
> > > > > 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer
> } }
> > > > > 2013-10-29 09:22:51,802 DEBUG
> > > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > Zone
> > > > 1
> > > > > is ready to launch secondary storage VM
> > > > > 2013-10-29 09:22:51,807 INFO
> > > > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > > (secstorage-1:null)
> > > > > No running secondary storage vms found in datacenter id=1, starting
> > one
> > > > > 2013-10-29 09:22:51,814 INFO
> > > > > [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null)
> > No
> > > > > stopped secondary storage vm is available, need to allocate a new
> > > > secondary
> > > > > storage vm
> > > > > 2013-10-29 09:22:51,817 DEBUG
> > > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > > Assign
> > > > > secondary storage vm from a newly started instance for request from
> > > data
> > > > > center : 1
> > > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Found existing network configuration for
> offering
> > > > > [Network Offering
> > [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> > > > > Ntwk[204|Guest|6]
> > > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Releasing lock for
> > > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Found existing network configuration for
> offering
> > > > > [Network Offering [3-Control-System-Control-Network]:
> > > Ntwk[202|Control|3]
> > > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Releasing lock for
> > > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > > 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Found existing network configuration for
> offering
> > > > > [Network Offering [2-Management-System-Management-Network]:
> > > > > Ntwk[201|Management|2]
> > > > > 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Releasing lock for
> > > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Found existing network configuration for
> offering
> > > > > [Network Offering [4-Storage-System-Storage-Network]:
> > > Ntwk[203|Storage|4]
> > > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Releasing lock for
> > > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > > 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Allocating entries for VM:
> > > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Allocating nics for
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Allocating nic for vm
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > in network Ntwk[204|Guest|6] with requested profile
> > > > > NicProfile[0-0-null-null-null
> > > > > 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Allocating nic for vm
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > in network Ntwk[202|Control|3] with requested profile null
> > > > > 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Allocating nic for vm
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > in network Ntwk[201|Management|2] with requested profile null
> > > > > 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> > > > > (secstorage-1:null) Allocating nic for vm
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > in network Ntwk[203|Storage|4] with requested profile null
> > > > > 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Allocating disks for
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Allocation completed for VM:
> > > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > > 2013-10-29 09:22:51,925 DEBUG
> > > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > > (secstorage-1:null) received secondary storage vm alert
> > > > > 2013-10-29 09:22:51,925 DEBUG
> > > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > > (secstorage-1:null) New secondary storage vm created, zone:
> > > > EastProdZone1,
> > > > > secStorageVm: s-4930-VM, public IP: null, private IP: null
> > > > > 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > > (secstorage-1:null) VM state transitted from :Stopped to Starting
> > with
> > > > > event: StartRequestedvm's original host id: null new host id: null
> > host
> > > > id
> > > > > before state transition: null
> > > > > 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Successfully transitioned to start state for
> > > > > VM[SecondaryStorageVm|s-4930-VM] reservation id =
> > > > > 3b2435a0-9e32-4777-a42f-a766a375bc1b
> > > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> > null
> > > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > > 2013-10-29 09:22:51,945 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > > 2013-10-29 09:22:51,946 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) DeploymentPlanner allocation algorithm:
> > > > >
> > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > > 2013-10-29 09:22:51,946 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) Trying to allocate a host and storage pools
> from
> > > > dc:1,
> > > > > pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> > > > > 2013-10-29 09:22:51,946 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) Is ROOT volume READY (pool already allocated)?:
> > No
> > > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) Searching all possible resources under this
> > Zone: 1
> > > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) Listing clusters in order of aggregate
> capacity,
> > > that
> > > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > > Zone: 1
> > > > > 2013-10-29 09:22:51,958 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > > 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> > > pod:1
> > > > > cluster:1
> > > > > 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0
> hosts
> > > to
> > > > > check for allocation: []
> > > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for
> allocation
> > > > after
> > > > > prioritization: []
> > > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > > Ram=256
> > > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > > > suitable hosts
> > > > > 2013-10-29 09:22:51,965 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) No suitable hosts found
> > > > > 2013-10-29 09:22:51,965 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) No suitable hosts found under this Cluster: 1
> > > > > 2013-10-29 09:22:51,968 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (secstorage-1:null) Could not find suitable Deployment Destination
> > for
> > > > > this VM under any clusters, returning.
> > > > > 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) Searching all possible resources under this
> > Zone: 1
> > > > > 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) Listing clusters in order of aggregate
> capacity,
> > > that
> > > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > > Zone: 1
> > > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) Removing from the clusterId list these clusters
> > > from
> > > > > avoid set: [1]
> > > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (secstorage-1:null) No clusters found after removing disabled
> > clusters
> > > > and
> > > > > clusters in avoid list, returning.
> > > > > 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > > (secstorage-1:null) VM state transitted from :Starting to Stopped
> > with
> > > > > event: OperationFailedvm's original host id: null new host id: null
> > > host
> > > > > id before state transition: null
> > > > > 2013-10-29 09:22:51,985 WARN
> > > > > [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null)
> > > > > Exception while trying to start secondary storage vm
> > > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > > create
> > > > > a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> > > > > com.cloud.dc.DataCenter; id=1
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > > > > at
> > > > >
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> > > > > at
> > > > >
> > >
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > > > at
> > > > >
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > > > at
> > > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > > at
> > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > > at java.lang.Thread.run(Thread.java:744)
> > > > > 2013-10-29 09:22:51,987 INFO
> > > > > [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null)
> > > > > Unable to start secondary storage vm for standby capacity,
> > secStorageVm
> > > > vm
> > > > > Id :
> > > > > 4930, will recycle it and start a new one
> > > > > 2013-10-29 09:22:51,987 INFO
> > > > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > > (secstorage-1:null)
> > > > > Primary secondary storage is not even started, wait until next turn
> > > > > 2013-10-29 09:22:52,009 DEBUG
> > > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > > (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> > > > > 2013-10-29 09:22:52,009 DEBUG
> > > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > > (consoleproxy-1:null) Expand console proxy standby capacity for
> zone
> > > > > EastProdZone1
> > > > > 2013-10-29 09:22:52,012 INFO
> > > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > > (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> > > > > running pool. proxy vm id : 4271
> > > > > 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > > (consoleproxy-1:null) VM state transitted from :Stopped to Starting
> > > with
> > > > > event: StartRequestedvm's original host id: null new host id: null
> > host
> > > > id
> > > > > before state transition: null
> > > > > 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (consoleproxy-1:null) Successfully transitioned to start state for
> > > > > VM[ConsoleProxy|v-4271-VM] reservation id =
> > > > > 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> > > > > 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and
> podId:
> > > null
> > > > > 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> > hosts:
> > > > null
> > > > > 2013-10-29 09:22:52,023 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> > hosts:
> > > > null
> > > > > 2013-10-29 09:22:52,024 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> > > > >
> > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > > 2013-10-29 09:22:52,024 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) Trying to allocate a host and storage pools
> > from
> > > > > dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
> > > > 1073741824
> > > > > 2013-10-29 09:22:52,024 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) Is ROOT volume READY (pool already
> allocated)?:
> > > No
> > > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) Searching all possible resources under this
> > > Zone: 1
> > > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> > capacity,
> > > > > that have (atleast one host with) enough CPU and RAM capacity under
> > > this
> > > > > Zone: 1
> > > > > 2013-10-29 09:22:52,031 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > > 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc:
> 1
> > > > pod:1
> > > > > cluster:1
> > > > > 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0
> > hosts
> > > > to
> > > > > check for allocation: []
> > > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for
> > allocation
> > > > > after prioritization: []
> > > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > > > > Ram=1024
> > > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > > (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning
> 0
> > > > > suitable hosts
> > > > > 2013-10-29 09:22:52,034 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) No suitable hosts found
> > > > > 2013-10-29 09:22:52,034 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> > > > > 2013-10-29 09:22:52,036 DEBUG
> > > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > > (consoleproxy-1:null) Could not find suitable Deployment
> Destination
> > > for
> > > > > this VM under any clusters, returning.
> > > > > 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) Searching all possible resources under this
> > > Zone: 1
> > > > > 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> > capacity,
> > > > > that have (atleast one host with) enough CPU and RAM capacity under
> > > this
> > > > > Zone: 1
> > > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) Removing from the clusterId list these
> clusters
> > > > from
> > > > > avoid set: [1]
> > > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > > (consoleproxy-1:null) No clusters found after removing disabled
> > > clusters
> > > > > and clusters in avoid list, returning.
> > > > > 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > > (consoleproxy-1:null) VM state transitted from :Starting to Stopped
> > > with
> > > > > event: OperationFailedvm's original host id: null new host id: null
> > > host
> > > > > id before state transition: null
> > > > > 2013-10-29 09:22:52,045 WARN
> > > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > > (consoleproxy-1:null) Exception while trying to start console proxy
> > > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > > create
> > > > > a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> > > > > com.cloud.dc.DataCenter; id=1
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> > > > > at
> > > > >
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> > > > > at
> > > > >
> > >
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > > > at
> > > > >
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > > > at
> > > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > > at
> > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > > at java.lang.Thread.run(Thread.java:744)
> > > > >
> > > > >
> > > > > What am I missing? How can I get these two system VMs online?
> > > > >
> > > > > Best Regards,
> > > > >
> > > > >
> > > > >
> > > > > Adam Scarcella
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com>
> > > wrote:
> > > > >
> > > > > > It's worth a shot. I've gone through the db manually, and changed
> > all
> > > > > > the places where I could see the old IPs, but your way is much
> more
> > > > > efficient.
> > > > > > Doing it now and will report back soon. Thanks.
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Adam Scarcella
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <
> > > > andrija.panic@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> one of the crazy tries:
> > > > > >> backup your database, search for all IP addresses inside SQL
> file,
> > > > > >> map old to new IP addresses (old1 --> new1, old2 -->new2...) and
> > > then
> > > > > >> do search/replace each old one with new one.
> > > > > >> This is for netowrking/ip addresses (guest, public,etc), same
> for
> > > NFS
> > > > > >> serves, same for NFS host, etc...
> > > > > >>
> > > > > >> Not sure if that is going to make you UP again, but...
> > > > > >>
> > > > > >>
> > > > > >> On 28 October 2013 22:59, Adam <ad...@gmail.com>
> wrote:
> > > > > >>
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > I know what I'm about to ask may seem like crazy talk, but I
> > have
> > > a
> > > > > >> > situation at work where our CloudStack "box" was moved to
> > another
> > > > > >> network.
> > > > > >> > This single server is a self-contained cloud with everything
> on
> > > one
> > > > > box.
> > > > > >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
> > > > > >> > primary
> > > > > >> and
> > > > > >> > secondary storage mounts. It's a very basic configuration, but
> > my
> > > > > >> company
> > > > > >> > moved from one town to another, and the IT dept decided to
> move
> > > our
> > > > > >> server
> > > > > >> > to a different switch and subnet without asking. So now
> > literally
> > > > > >> > every single IP address of any importance has been changed.
> > > > > >> >
> > > > > >> > I've been trying in vain to get this CS up and running again,
> > but
> > > > > >> > it
> > > > > >> seems
> > > > > >> > like everything has changed including the guids, or at least
> > that
> > > > > >> > seems
> > > > > >> to
> > > > > >> > be the case since I cannot match up guids from the database to
> > > > > >> > guids on
> > > > > >> the
> > > > > >> > primary/secondary NFS.
> > > > > >> >
> > > > > >> > I'm sure if each piece were separate and moved one piece at a
> > > time,
> > > > > >> > it would be fairly straightforward, but since this box is
> > > > > >> > self-contained
> > > > > >> and
> > > > > >> > everything changed at the same time, it is proving to be very
> > > > > difficult.
> > > > > >> >
> > > > > >> > Does anyone have any suggestions for me to attempt to recover
> my
> > > CS?
> > > > > >> >
> > > > > >> > Best Regards,
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > Adam Scarcella
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> Andrija Panić
> > > > > >> --------------------------------------
> > > > > >> http://admintweets.com
> > > > > >> --------------------------------------
> > > > > >>
> > > > > >
> > > > > >
> > > > > This email and any attachments to it may be confidential and are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed. Any
> > views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you are
> > not
> > > > the
> > > > > intended recipient of this email, you must neither take any action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please contact
> the
> > > > sender
> > > > > if you believe you have received this email in error. Shape Blue
> Ltd
> > > is a
> > > > > company incorporated in England & Wales. ShapeBlue Services India
> LLP
> > > is
> > > > a
> > > > > company incorporated in India and is operated under license from
> > Shape
> > > > Blue
> > > > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
> in
> > > > Brasil
> > > > > and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > > > > registered trademark.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > > --------------------------------------
> > > http://admintweets.com
> > > --------------------------------------
> > >
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Andrija Panic <an...@gmail.com>.
my bad for secondary storage - there is no maintance mode for that :)
I was just refering to line:
host=10.97.38.10 - which is hopefully new IP address :)
Since you got Server capactity issues, I suggest folowing, as temp solution:
Check CPU alocation, RAM alocation, and increase overprovisioning factor at
POD level (at least this is in 4.2, or only in global config as was in
4.0.0), so if you are running out of CPU/RAM resopurces, you might be able
to fix this this way.
I had this exact type of problem (SSVM not starting), when upgraded from
4,0 to 4.2,because of bad SSVM template - I had to download
http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2(previously
I had very old, which came with Cs 4.0.0) - but if you did not
upgrade your CS, then I suppose this is not the case...
Best
On 29 October 2013 16:42, Adam <ad...@gmail.com> wrote:
> Hi Andrija,
>
> Thanks for your reply.
>
> Here is the output of my agent.properties file (which looks correct to me):
>
> [root@cs-east-prod1 ~]# cat /etc/cloudstack/agent/agent.properties
> #Storage
> #Tue Oct 29 10:15:49 GMT-05:00 2013
> guest.network.device=cloudbr0
> workers=5
> private.network.device=cloudbr0
> port=8250
> resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
> pod=1
> zone=1
> guid=db24e08f-ac6e-3974-b223-274d49af52d0
> cluster=1
> public.network.device=cloudbr0
> local.storage.uuid=296f0404-9658-4269-bc24-64b1247d5e3f
> domr.scripts.dir=scripts/network/domr/kvm
> host=10.97.38.10
> LibvirtComputingResource.id=10
>
> [root@cs-east-prod1 ~]# ifconfig cloudbr0
> cloudbr0 Link encap:Ethernet HWaddr AC:16:2D:84:D4:24
> inet addr:10.97.38.10 Bcast:10.97.38.255 Mask:255.255.255.0
> inet6 addr: fe80::ae16:2dff:fe84:d424/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:10216 errors:0 dropped:0 overruns:0 frame:0
> TX packets:10343 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:1605339 (1.5 MiB) TX bytes:24242332 (23.1 MiB)
>
>
> I have 4.5 TB free on my 5.7 TB RAID so I haven't even come close to the
> "pool.storage.allocated.capacity.disablethreshold" or
> "pool.storage.capacity.disablethreshold" of "0.85"
>
> [root@cs-east-prod1 ~]# df -kh
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/vg_cloudstackuse-lv_root
> 5.7T 921G 4.5T 17% /
> tmpfs 64G 72K 64G 1% /dev/shm
> /dev/sda1 485M 118M 342M 26% /boot
> 10.97.38.10:/export/primary
> 5.7T 921G 4.5T 17%
> /mnt/910365e6-a298-37b8-904f-2b49198c8286
> [root@cs-east-prod1 ~]# cd /export/
> [root@cs-east-prod1 export]# ls -l
> total 8
> drwxrwxrwx. 3 root root 4096 Oct 27 19:18 primary
> drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondary
> [root@cs-east-prod1 export]# du -sh *
> 238G primary
> 110G secondary
> [root@cs-east-prod1 export]#
>
> I've manually destroyed the SSVM several times and yes CS does attempt to
> create a new one, but the same issue occurs. Regardless of whether I
> destroy the SSVM or if CS does it, it just keeps looping.
>
> I'm not exactly sure how to check if my secondary storage is in Maintenance
> Mode. I know I can enable Maintenance Mode of my Primary Storage, but how
> to check for the Secondary Storage?
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Tue, Oct 29, 2013 at 11:16 AM, Andrija Panic <andrija.panic@gmail.com
> >wrote:
>
> > I suppose, you can recheck the network, and also check the AGENT
> > configuration file, make changes
> > there: /etc/cloudstack/agent/agent.properties
> >
> > From logs:
> >
> > InsufficientServerCapacityException: Unable to create
> > > a deployment for VM.....
> >
> > I had this kind of error, when my storage pools were disabled once, and
> > also when I have reached "disable capacity treshold" - my storage pool
> was
> > over 85% full.
> >
> > So check if you secondaru storge is in maintance mode, or similar...
> >
> > Also, destroying SSVM will force CS into making new one...
> >
> >
> > On 29 October 2013 16:09, Adam <ad...@gmail.com> wrote:
> >
> > > Hi Geoff,
> > >
> > > Thanks for the reply, I was able to manually mount the secondary
> storage
> > > NFS:
> > >
> > > [root@cs-east-prod1 secondaryStorage]# hostname
> > > cs-east-prod1
> > > [root@cs-east-prod1 secondaryStorage]# hostname --fqdn
> > > cs-east-prod1.na.tibco.com
> > > [root@cs-east-prod1 secondaryStorage]#
> > > [root@cs-east-prod1 ~]# mkdir /secondaryStorage
> > > [root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
> > > /secondaryStorage
> > > [root@cs-east-prod1 ~]# cd /
> > > [root@cs-east-prod1 /]# ls -l
> > > total 120
> > > dr-xr-xr-x. 2 root root 4096 Oct 23 04:48 bin
> > > dr-xr-xr-x. 5 root root 3072 Oct 22 21:10 boot
> > > drwxr-xr-x. 10 root root 4096 Jun 25 10:23 cgroup
> > > drwxr-xr-x. 19 root root 3960 Oct 29 10:38 dev
> > > drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
> > > drwxrwxrwx. 4 root root 4096 Jul 3 20:10 export
> > > drwxr-xr-x. 3 root root 4096 Jul 3 19:14 home
> > > lrwxrwxrwx. 1 root root 25 Aug 28 15:46 Installers ->
> > > /var/www/html/Installers/
> > > lrwxrwxrwx. 1 root root 19 Jul 5 20:58 ISOs ->
> /var/www/html/ISOs/
> > > dr-xr-xr-x. 11 root root 4096 Jul 3 19:02 lib
> > > dr-xr-xr-x. 10 root root 12288 Oct 28 03:33 lib64
> > > drwx------. 2 root root 16384 Jul 3 18:44 lost+found
> > > drwxr-xr-x. 3 root root 4096 Oct 27 12:05 media
> > > drwxr-xr-x. 2 root root 0 Oct 29 10:08 misc
> > > drwxr-xr-x. 39 root root 4096 Sep 6 12:07 mnt
> > > drwxr-xr-x. 2 root root 0 Oct 29 10:08 net
> > > drwxr-xr-x. 3 root root 4096 Jul 3 19:05 opt
> > > drwxr-xr-x. 2 root root 4096 Oct 27 10:06 primarymount
> > > dr-xr-xr-x. 557 root root 0 Oct 29 10:08 proc
> > > dr-xr-x---. 35 root root 4096 Oct 29 10:09 root
> > > dr-xr-xr-x. 2 root root 12288 Oct 28 03:33 sbin
> > > drwxr-xr-x. 2 root root 4096 Oct 27 10:07 secondarymount
> > > drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondaryStorage
> > > drwxr-xr-x. 7 root root 0 Oct 29 10:08 selinux
> > > drwxr-xr-x. 2 root root 4096 Sep 23 2011 srv
> > > drwxr-xr-x. 13 root root 0 Oct 29 10:08 sys
> > > drwxrwxrwt. 16 root root 4096 Oct 29 11:07 tmp
> > > drwxr-xr-x. 13 root root 4096 Jul 3 18:55 usr
> > > drwxr-xr-x. 23 root root 4096 Jul 3 19:45 var
> > > [root@cs-east-prod1 /]# cd secondaryStorage/
> > > [root@cs-east-prod1 secondaryStorage]# ls -l
> > > total 12
> > > drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
> > > drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
> > > drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
> > > [root@cs-east-prod1 secondaryStorage]#
> > >
> > >
> > >
> > > Best Regards,
> > >
> > >
> > >
> > > Adam Scarcella
> > >
> > >
> > > On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
> > > geoff.higginbottom@shapeblue.com> wrote:
> > >
> > > > Adam,
> > > >
> > > > System VMs looping and failing to create is often caused by the Host
> > > being
> > > > unable to access Sec Storage.
> > > >
> > > > Do a manual mount of the Sec Storage NFS on the Host to confirm it
> can
> > > > access it.
> > > >
> > > > Regards
> > > >
> > > > Geoff Higginbottom
> > > >
> > > > D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
> > > >
> > > > geoff.higginbottom@shapeblue.com
> > > >
> > > > -----Original Message-----
> > > > From: Adam [mailto:adam.scarcella@gmail.com]
> > > > Sent: 29 October 2013 14:44
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: How to recover entire self contained CloudStack after
> > major
> > > > network changes?
> > > >
> > > > I exported the db as a *.sql file, did search/replace and that seemed
> > to
> > > > help a lot, but still not 100% out of the woods. Now I'm hitting the
> > > issue
> > > > where the System VMs (SecondaryStorageVm & ConsoleProxy) do not
> start.
> > > They
> > > > just keep looping from creation to being expunged.
> > > >
> > > > My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> > > > Primary Storage is online:
> > > >
> > > > [root@cs-east-prod1 ~]# df -kh
> > > > Filesystem Size Used Avail Use% Mounted on
> > > > /dev/mapper/vg_cloudstackuse-lv_root
> > > > 5.7T 921G 4.5T 17% /
> > > > tmpfs 64G 72K 64G 1% /dev/shm
> > > > /dev/sda1 485M 118M 342M 26% /boot
> > > > 10.97.38.10:/export/primary
> > > > 5.7T 921G 4.5T 17%
> > > > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> > > >
> > > > Here is an excerpt from my most recent log:
> > > >
> > > > 2013-10-29 09:22:23,389 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > 2013-10-29 09:22:23,391 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > 2013-10-29 09:22:23,441 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > 2013-10-29 09:22:23,443 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> > > > (AgentManager-Handler-5:null) Ping from 10
> > > > 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-2:null) VmStatsCollector is running...
> > > > 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-3:null) StorageCollector is running...
> > > > 2013-10-29 09:22:38,716 INFO
> > [storage.endpoint.DefaultEndPointSelector]
> > > > (StatsCollector-3:null) No running ssvm is found, so command will be
> > sent
> > > > to LocalHostEndPoint
> > > > 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-1:null) HostStatsCollector is running...
> > > > 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> > > > (StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
> > > > 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > > > 2013-10-29 09:22:51,802 DEBUG
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Zone
> > > 1
> > > > is ready to launch secondary storage VM
> > > > 2013-10-29 09:22:51,807 INFO
> > > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > (secstorage-1:null)
> > > > No running secondary storage vms found in datacenter id=1, starting
> one
> > > > 2013-10-29 09:22:51,814 INFO
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> No
> > > > stopped secondary storage vm is available, need to allocate a new
> > > secondary
> > > > storage vm
> > > > 2013-10-29 09:22:51,817 DEBUG
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > Assign
> > > > secondary storage vm from a newly started instance for request from
> > data
> > > > center : 1
> > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering
> [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> > > > Ntwk[204|Guest|6]
> > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [3-Control-System-Control-Network]:
> > Ntwk[202|Control|3]
> > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [2-Management-System-Management-Network]:
> > > > Ntwk[201|Management|2]
> > > > 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [4-Storage-System-Storage-Network]:
> > Ntwk[203|Storage|4]
> > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating entries for VM:
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating nics for
> > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[204|Guest|6] with requested profile
> > > > NicProfile[0-0-null-null-null
> > > > 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[202|Control|3] with requested profile null
> > > > 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[201|Management|2] with requested profile null
> > > > 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[203|Storage|4] with requested profile null
> > > > 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating disks for
> > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocation completed for VM:
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,925 DEBUG
> > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > (secstorage-1:null) received secondary storage vm alert
> > > > 2013-10-29 09:22:51,925 DEBUG
> > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > (secstorage-1:null) New secondary storage vm created, zone:
> > > EastProdZone1,
> > > > secStorageVm: s-4930-VM, public IP: null, private IP: null
> > > > 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (secstorage-1:null) VM state transitted from :Stopped to Starting
> with
> > > > event: StartRequestedvm's original host id: null new host id: null
> host
> > > id
> > > > before state transition: null
> > > > 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Successfully transitioned to start state for
> > > > VM[SecondaryStorageVm|s-4930-VM] reservation id =
> > > > 3b2435a0-9e32-4777-a42f-a766a375bc1b
> > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> null
> > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > > 2013-10-29 09:22:51,945 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) DeploymentPlanner allocation algorithm:
> > > >
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Trying to allocate a host and storage pools from
> > > dc:1,
> > > > pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Is ROOT volume READY (pool already allocated)?:
> No
> > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Searching all possible resources under this
> Zone: 1
> > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> > that
> > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > Zone: 1
> > > > 2013-10-29 09:22:51,958 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> > pod:1
> > > > cluster:1
> > > > 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts
> > to
> > > > check for allocation: []
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> > > after
> > > > prioritization: []
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > Ram=256
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > > suitable hosts
> > > > 2013-10-29 09:22:51,965 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) No suitable hosts found
> > > > 2013-10-29 09:22:51,965 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) No suitable hosts found under this Cluster: 1
> > > > 2013-10-29 09:22:51,968 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Could not find suitable Deployment Destination
> for
> > > > this VM under any clusters, returning.
> > > > 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Searching all possible resources under this
> Zone: 1
> > > > 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> > that
> > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > Zone: 1
> > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Removing from the clusterId list these clusters
> > from
> > > > avoid set: [1]
> > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) No clusters found after removing disabled
> clusters
> > > and
> > > > clusters in avoid list, returning.
> > > > 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (secstorage-1:null) VM state transitted from :Starting to Stopped
> with
> > > > event: OperationFailedvm's original host id: null new host id: null
> > host
> > > > id before state transition: null
> > > > 2013-10-29 09:22:51,985 WARN
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > > Exception while trying to start secondary storage vm
> > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > create
> > > > a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> > > > com.cloud.dc.DataCenter; id=1
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > > > at
> > > >
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > > at
> > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > 2013-10-29 09:22:51,987 INFO
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > > Unable to start secondary storage vm for standby capacity,
> secStorageVm
> > > vm
> > > > Id :
> > > > 4930, will recycle it and start a new one
> > > > 2013-10-29 09:22:51,987 INFO
> > > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > (secstorage-1:null)
> > > > Primary secondary storage is not even started, wait until next turn
> > > > 2013-10-29 09:22:52,009 DEBUG
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> > > > 2013-10-29 09:22:52,009 DEBUG
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Expand console proxy standby capacity for zone
> > > > EastProdZone1
> > > > 2013-10-29 09:22:52,012 INFO
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> > > > running pool. proxy vm id : 4271
> > > > 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (consoleproxy-1:null) VM state transitted from :Stopped to Starting
> > with
> > > > event: StartRequestedvm's original host id: null new host id: null
> host
> > > id
> > > > before state transition: null
> > > > 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Successfully transitioned to start state for
> > > > VM[ConsoleProxy|v-4271-VM] reservation id =
> > > > 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> > > > 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> > null
> > > > 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > 2013-10-29 09:22:52,023 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> > > >
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Trying to allocate a host and storage pools
> from
> > > > dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
> > > 1073741824
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?:
> > No
> > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Searching all possible resources under this
> > Zone: 1
> > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> capacity,
> > > > that have (atleast one host with) enough CPU and RAM capacity under
> > this
> > > > Zone: 1
> > > > 2013-10-29 09:22:52,031 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> > > pod:1
> > > > cluster:1
> > > > 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0
> hosts
> > > to
> > > > check for allocation: []
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for
> allocation
> > > > after prioritization: []
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > > > Ram=1024
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > > suitable hosts
> > > > 2013-10-29 09:22:52,034 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) No suitable hosts found
> > > > 2013-10-29 09:22:52,034 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> > > > 2013-10-29 09:22:52,036 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Could not find suitable Deployment Destination
> > for
> > > > this VM under any clusters, returning.
> > > > 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Searching all possible resources under this
> > Zone: 1
> > > > 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> capacity,
> > > > that have (atleast one host with) enough CPU and RAM capacity under
> > this
> > > > Zone: 1
> > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Removing from the clusterId list these clusters
> > > from
> > > > avoid set: [1]
> > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) No clusters found after removing disabled
> > clusters
> > > > and clusters in avoid list, returning.
> > > > 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (consoleproxy-1:null) VM state transitted from :Starting to Stopped
> > with
> > > > event: OperationFailedvm's original host id: null new host id: null
> > host
> > > > id before state transition: null
> > > > 2013-10-29 09:22:52,045 WARN
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Exception while trying to start console proxy
> > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > create
> > > > a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> > > > com.cloud.dc.DataCenter; id=1
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> > > > at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> > > > at
> > > >
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > > at
> > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > >
> > > >
> > > > What am I missing? How can I get these two system VMs online?
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com>
> > wrote:
> > > >
> > > > > It's worth a shot. I've gone through the db manually, and changed
> all
> > > > > the places where I could see the old IPs, but your way is much more
> > > > efficient.
> > > > > Doing it now and will report back soon. Thanks.
> > > > >
> > > > > Best Regards,
> > > > >
> > > > >
> > > > >
> > > > > Adam Scarcella
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <
> > > andrija.panic@gmail.com
> > > > >wrote:
> > > > >
> > > > >> one of the crazy tries:
> > > > >> backup your database, search for all IP addresses inside SQL file,
> > > > >> map old to new IP addresses (old1 --> new1, old2 -->new2...) and
> > then
> > > > >> do search/replace each old one with new one.
> > > > >> This is for netowrking/ip addresses (guest, public,etc), same for
> > NFS
> > > > >> serves, same for NFS host, etc...
> > > > >>
> > > > >> Not sure if that is going to make you UP again, but...
> > > > >>
> > > > >>
> > > > >> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > I know what I'm about to ask may seem like crazy talk, but I
> have
> > a
> > > > >> > situation at work where our CloudStack "box" was moved to
> another
> > > > >> network.
> > > > >> > This single server is a self-contained cloud with everything on
> > one
> > > > box.
> > > > >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
> > > > >> > primary
> > > > >> and
> > > > >> > secondary storage mounts. It's a very basic configuration, but
> my
> > > > >> company
> > > > >> > moved from one town to another, and the IT dept decided to move
> > our
> > > > >> server
> > > > >> > to a different switch and subnet without asking. So now
> literally
> > > > >> > every single IP address of any importance has been changed.
> > > > >> >
> > > > >> > I've been trying in vain to get this CS up and running again,
> but
> > > > >> > it
> > > > >> seems
> > > > >> > like everything has changed including the guids, or at least
> that
> > > > >> > seems
> > > > >> to
> > > > >> > be the case since I cannot match up guids from the database to
> > > > >> > guids on
> > > > >> the
> > > > >> > primary/secondary NFS.
> > > > >> >
> > > > >> > I'm sure if each piece were separate and moved one piece at a
> > time,
> > > > >> > it would be fairly straightforward, but since this box is
> > > > >> > self-contained
> > > > >> and
> > > > >> > everything changed at the same time, it is proving to be very
> > > > difficult.
> > > > >> >
> > > > >> > Does anyone have any suggestions for me to attempt to recover my
> > CS?
> > > > >> >
> > > > >> > Best Regards,
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Adam Scarcella
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Andrija Panić
> > > > >> --------------------------------------
> > > > >> http://admintweets.com
> > > > >> --------------------------------------
> > > > >>
> > > > >
> > > > >
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed. Any
> views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you are
> not
> > > the
> > > > intended recipient of this email, you must neither take any action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact the
> > > sender
> > > > if you believe you have received this email in error. Shape Blue Ltd
> > is a
> > > > company incorporated in England & Wales. ShapeBlue Services India LLP
> > is
> > > a
> > > > company incorporated in India and is operated under license from
> Shape
> > > Blue
> > > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > > Brasil
> > > > and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > > > registered trademark.
> > > >
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> > --------------------------------------
> > http://admintweets.com
> > --------------------------------------
> >
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Adam <ad...@gmail.com>.
Hi Andrija,
Thanks for your reply.
Here is the output of my agent.properties file (which looks correct to me):
[root@cs-east-prod1 ~]# cat /etc/cloudstack/agent/agent.properties
#Storage
#Tue Oct 29 10:15:49 GMT-05:00 2013
guest.network.device=cloudbr0
workers=5
private.network.device=cloudbr0
port=8250
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
pod=1
zone=1
guid=db24e08f-ac6e-3974-b223-274d49af52d0
cluster=1
public.network.device=cloudbr0
local.storage.uuid=296f0404-9658-4269-bc24-64b1247d5e3f
domr.scripts.dir=scripts/network/domr/kvm
host=10.97.38.10
LibvirtComputingResource.id=10
[root@cs-east-prod1 ~]# ifconfig cloudbr0
cloudbr0 Link encap:Ethernet HWaddr AC:16:2D:84:D4:24
inet addr:10.97.38.10 Bcast:10.97.38.255 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:fe84:d424/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10216 errors:0 dropped:0 overruns:0 frame:0
TX packets:10343 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1605339 (1.5 MiB) TX bytes:24242332 (23.1 MiB)
I have 4.5 TB free on my 5.7 TB RAID so I haven't even come close to the
"pool.storage.allocated.capacity.disablethreshold" or
"pool.storage.capacity.disablethreshold" of "0.85"
[root@cs-east-prod1 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_cloudstackuse-lv_root
5.7T 921G 4.5T 17% /
tmpfs 64G 72K 64G 1% /dev/shm
/dev/sda1 485M 118M 342M 26% /boot
10.97.38.10:/export/primary
5.7T 921G 4.5T 17%
/mnt/910365e6-a298-37b8-904f-2b49198c8286
[root@cs-east-prod1 ~]# cd /export/
[root@cs-east-prod1 export]# ls -l
total 8
drwxrwxrwx. 3 root root 4096 Oct 27 19:18 primary
drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondary
[root@cs-east-prod1 export]# du -sh *
238G primary
110G secondary
[root@cs-east-prod1 export]#
I've manually destroyed the SSVM several times and yes CS does attempt to
create a new one, but the same issue occurs. Regardless of whether I
destroy the SSVM or if CS does it, it just keeps looping.
I'm not exactly sure how to check if my secondary storage is in Maintenance
Mode. I know I can enable Maintenance Mode of my Primary Storage, but how
to check for the Secondary Storage?
Best Regards,
Adam Scarcella
On Tue, Oct 29, 2013 at 11:16 AM, Andrija Panic <an...@gmail.com>wrote:
> I suppose, you can recheck the network, and also check the AGENT
> configuration file, make changes
> there: /etc/cloudstack/agent/agent.properties
>
> From logs:
>
> InsufficientServerCapacityException: Unable to create
> > a deployment for VM.....
>
> I had this kind of error, when my storage pools were disabled once, and
> also when I have reached "disable capacity treshold" - my storage pool was
> over 85% full.
>
> So check if you secondaru storge is in maintance mode, or similar...
>
> Also, destroying SSVM will force CS into making new one...
>
>
> On 29 October 2013 16:09, Adam <ad...@gmail.com> wrote:
>
> > Hi Geoff,
> >
> > Thanks for the reply, I was able to manually mount the secondary storage
> > NFS:
> >
> > [root@cs-east-prod1 secondaryStorage]# hostname
> > cs-east-prod1
> > [root@cs-east-prod1 secondaryStorage]# hostname --fqdn
> > cs-east-prod1.na.tibco.com
> > [root@cs-east-prod1 secondaryStorage]#
> > [root@cs-east-prod1 ~]# mkdir /secondaryStorage
> > [root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
> > /secondaryStorage
> > [root@cs-east-prod1 ~]# cd /
> > [root@cs-east-prod1 /]# ls -l
> > total 120
> > dr-xr-xr-x. 2 root root 4096 Oct 23 04:48 bin
> > dr-xr-xr-x. 5 root root 3072 Oct 22 21:10 boot
> > drwxr-xr-x. 10 root root 4096 Jun 25 10:23 cgroup
> > drwxr-xr-x. 19 root root 3960 Oct 29 10:38 dev
> > drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
> > drwxrwxrwx. 4 root root 4096 Jul 3 20:10 export
> > drwxr-xr-x. 3 root root 4096 Jul 3 19:14 home
> > lrwxrwxrwx. 1 root root 25 Aug 28 15:46 Installers ->
> > /var/www/html/Installers/
> > lrwxrwxrwx. 1 root root 19 Jul 5 20:58 ISOs -> /var/www/html/ISOs/
> > dr-xr-xr-x. 11 root root 4096 Jul 3 19:02 lib
> > dr-xr-xr-x. 10 root root 12288 Oct 28 03:33 lib64
> > drwx------. 2 root root 16384 Jul 3 18:44 lost+found
> > drwxr-xr-x. 3 root root 4096 Oct 27 12:05 media
> > drwxr-xr-x. 2 root root 0 Oct 29 10:08 misc
> > drwxr-xr-x. 39 root root 4096 Sep 6 12:07 mnt
> > drwxr-xr-x. 2 root root 0 Oct 29 10:08 net
> > drwxr-xr-x. 3 root root 4096 Jul 3 19:05 opt
> > drwxr-xr-x. 2 root root 4096 Oct 27 10:06 primarymount
> > dr-xr-xr-x. 557 root root 0 Oct 29 10:08 proc
> > dr-xr-x---. 35 root root 4096 Oct 29 10:09 root
> > dr-xr-xr-x. 2 root root 12288 Oct 28 03:33 sbin
> > drwxr-xr-x. 2 root root 4096 Oct 27 10:07 secondarymount
> > drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondaryStorage
> > drwxr-xr-x. 7 root root 0 Oct 29 10:08 selinux
> > drwxr-xr-x. 2 root root 4096 Sep 23 2011 srv
> > drwxr-xr-x. 13 root root 0 Oct 29 10:08 sys
> > drwxrwxrwt. 16 root root 4096 Oct 29 11:07 tmp
> > drwxr-xr-x. 13 root root 4096 Jul 3 18:55 usr
> > drwxr-xr-x. 23 root root 4096 Jul 3 19:45 var
> > [root@cs-east-prod1 /]# cd secondaryStorage/
> > [root@cs-east-prod1 secondaryStorage]# ls -l
> > total 12
> > drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
> > drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
> > drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
> > [root@cs-east-prod1 secondaryStorage]#
> >
> >
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
> > geoff.higginbottom@shapeblue.com> wrote:
> >
> > > Adam,
> > >
> > > System VMs looping and failing to create is often caused by the Host
> > being
> > > unable to access Sec Storage.
> > >
> > > Do a manual mount of the Sec Storage NFS on the Host to confirm it can
> > > access it.
> > >
> > > Regards
> > >
> > > Geoff Higginbottom
> > >
> > > D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
> > >
> > > geoff.higginbottom@shapeblue.com
> > >
> > > -----Original Message-----
> > > From: Adam [mailto:adam.scarcella@gmail.com]
> > > Sent: 29 October 2013 14:44
> > > To: users@cloudstack.apache.org
> > > Subject: Re: How to recover entire self contained CloudStack after
> major
> > > network changes?
> > >
> > > I exported the db as a *.sql file, did search/replace and that seemed
> to
> > > help a lot, but still not 100% out of the woods. Now I'm hitting the
> > issue
> > > where the System VMs (SecondaryStorageVm & ConsoleProxy) do not start.
> > They
> > > just keep looping from creation to being expunged.
> > >
> > > My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> > > Primary Storage is online:
> > >
> > > [root@cs-east-prod1 ~]# df -kh
> > > Filesystem Size Used Avail Use% Mounted on
> > > /dev/mapper/vg_cloudstackuse-lv_root
> > > 5.7T 921G 4.5T 17% /
> > > tmpfs 64G 72K 64G 1% /dev/shm
> > > /dev/sda1 485M 118M 342M 26% /boot
> > > 10.97.38.10:/export/primary
> > > 5.7T 921G 4.5T 17%
> > > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> > >
> > > Here is an excerpt from my most recent log:
> > >
> > > 2013-10-29 09:22:23,389 DEBUG
> > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > 2013-10-29 09:22:23,391 DEBUG
> > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > 2013-10-29 09:22:23,441 DEBUG
> > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > 2013-10-29 09:22:23,443 DEBUG
> > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> > > (AgentManager-Handler-5:null) Ping from 10
> > > 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> > > (StatsCollector-2:null) VmStatsCollector is running...
> > > 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> > > (StatsCollector-3:null) StorageCollector is running...
> > > 2013-10-29 09:22:38,716 INFO
> [storage.endpoint.DefaultEndPointSelector]
> > > (StatsCollector-3:null) No running ssvm is found, so command will be
> sent
> > > to LocalHostEndPoint
> > > 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> > > (StatsCollector-1:null) HostStatsCollector is running...
> > > 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> > > (StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
> > > 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > > 2013-10-29 09:22:51,802 DEBUG
> > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Zone
> > 1
> > > is ready to launch secondary storage VM
> > > 2013-10-29 09:22:51,807 INFO
> > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > (secstorage-1:null)
> > > No running secondary storage vms found in datacenter id=1, starting one
> > > 2013-10-29 09:22:51,814 INFO
> > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No
> > > stopped secondary storage vm is available, need to allocate a new
> > secondary
> > > storage vm
> > > 2013-10-29 09:22:51,817 DEBUG
> > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Assign
> > > secondary storage vm from a newly started instance for request from
> data
> > > center : 1
> > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Found existing network configuration for offering
> > > [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> > > Ntwk[204|Guest|6]
> > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Releasing lock for
> > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Found existing network configuration for offering
> > > [Network Offering [3-Control-System-Control-Network]:
> Ntwk[202|Control|3]
> > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Releasing lock for
> > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Found existing network configuration for offering
> > > [Network Offering [2-Management-System-Management-Network]:
> > > Ntwk[201|Management|2]
> > > 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Releasing lock for
> > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Found existing network configuration for offering
> > > [Network Offering [4-Storage-System-Storage-Network]:
> Ntwk[203|Storage|4]
> > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Releasing lock for
> > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Allocating entries for VM:
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Allocating nics for
> VM[SecondaryStorageVm|s-4930-VM]
> > > 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Allocating nic for vm
> > VM[SecondaryStorageVm|s-4930-VM]
> > > in network Ntwk[204|Guest|6] with requested profile
> > > NicProfile[0-0-null-null-null
> > > 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Allocating nic for vm
> > VM[SecondaryStorageVm|s-4930-VM]
> > > in network Ntwk[202|Control|3] with requested profile null
> > > 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Allocating nic for vm
> > VM[SecondaryStorageVm|s-4930-VM]
> > > in network Ntwk[201|Management|2] with requested profile null
> > > 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> > > (secstorage-1:null) Allocating nic for vm
> > VM[SecondaryStorageVm|s-4930-VM]
> > > in network Ntwk[203|Storage|4] with requested profile null
> > > 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Allocating disks for
> VM[SecondaryStorageVm|s-4930-VM]
> > > 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Allocation completed for VM:
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > 2013-10-29 09:22:51,925 DEBUG
> > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > (secstorage-1:null) received secondary storage vm alert
> > > 2013-10-29 09:22:51,925 DEBUG
> > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > (secstorage-1:null) New secondary storage vm created, zone:
> > EastProdZone1,
> > > secStorageVm: s-4930-VM, public IP: null, private IP: null
> > > 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > (secstorage-1:null) VM state transitted from :Stopped to Starting with
> > > event: StartRequestedvm's original host id: null new host id: null host
> > id
> > > before state transition: null
> > > 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Successfully transitioned to start state for
> > > VM[SecondaryStorageVm|s-4930-VM] reservation id =
> > > 3b2435a0-9e32-4777-a42f-a766a375bc1b
> > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
> > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> null
> > > 2013-10-29 09:22:51,945 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> null
> > > 2013-10-29 09:22:51,946 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) DeploymentPlanner allocation algorithm:
> > > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > 2013-10-29 09:22:51,946 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) Trying to allocate a host and storage pools from
> > dc:1,
> > > pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> > > 2013-10-29 09:22:51,946 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
> > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) Searching all possible resources under this Zone: 1
> > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> that
> > > have (atleast one host with) enough CPU and RAM capacity under this
> > Zone: 1
> > > 2013-10-29 09:22:51,958 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> > > (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> pod:1
> > > cluster:1
> > > 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> > > (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts
> to
> > > check for allocation: []
> > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> > after
> > > prioritization: []
> > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> Ram=256
> > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > suitable hosts
> > > 2013-10-29 09:22:51,965 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) No suitable hosts found
> > > 2013-10-29 09:22:51,965 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) No suitable hosts found under this Cluster: 1
> > > 2013-10-29 09:22:51,968 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (secstorage-1:null) Could not find suitable Deployment Destination for
> > > this VM under any clusters, returning.
> > > 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) Searching all possible resources under this Zone: 1
> > > 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> that
> > > have (atleast one host with) enough CPU and RAM capacity under this
> > Zone: 1
> > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) Removing from the clusterId list these clusters
> from
> > > avoid set: [1]
> > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (secstorage-1:null) No clusters found after removing disabled clusters
> > and
> > > clusters in avoid list, returning.
> > > 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > (secstorage-1:null) VM state transitted from :Starting to Stopped with
> > > event: OperationFailedvm's original host id: null new host id: null
> host
> > > id before state transition: null
> > > 2013-10-29 09:22:51,985 WARN
> > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > Exception while trying to start secondary storage vm
> > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create
> > > a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> > > com.cloud.dc.DataCenter; id=1
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > at
> > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> > > at
> > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> > > at
> > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> > > at
> > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > > at
> > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > > at
> > > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > at
> > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > 2013-10-29 09:22:51,987 INFO
> > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > Unable to start secondary storage vm for standby capacity, secStorageVm
> > vm
> > > Id :
> > > 4930, will recycle it and start a new one
> > > 2013-10-29 09:22:51,987 INFO
> > > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > (secstorage-1:null)
> > > Primary secondary storage is not even started, wait until next turn
> > > 2013-10-29 09:22:52,009 DEBUG
> > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> > > 2013-10-29 09:22:52,009 DEBUG
> > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:null) Expand console proxy standby capacity for zone
> > > EastProdZone1
> > > 2013-10-29 09:22:52,012 INFO
> > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> > > running pool. proxy vm id : 4271
> > > 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > (consoleproxy-1:null) VM state transitted from :Stopped to Starting
> with
> > > event: StartRequestedvm's original host id: null new host id: null host
> > id
> > > before state transition: null
> > > 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (consoleproxy-1:null) Successfully transitioned to start state for
> > > VM[ConsoleProxy|v-4271-VM] reservation id =
> > > 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> > > 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> null
> > > 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > 2013-10-29 09:22:52,023 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > 2013-10-29 09:22:52,024 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> > > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > 2013-10-29 09:22:52,024 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) Trying to allocate a host and storage pools from
> > > dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
> > 1073741824
> > > 2013-10-29 09:22:52,024 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?:
> No
> > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) Searching all possible resources under this
> Zone: 1
> > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> > > that have (atleast one host with) enough CPU and RAM capacity under
> this
> > > Zone: 1
> > > 2013-10-29 09:22:52,031 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> > pod:1
> > > cluster:1
> > > 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> > > (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts
> > to
> > > check for allocation: []
> > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> > > after prioritization: []
> > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > > Ram=1024
> > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > suitable hosts
> > > 2013-10-29 09:22:52,034 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) No suitable hosts found
> > > 2013-10-29 09:22:52,034 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> > > 2013-10-29 09:22:52,036 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > (consoleproxy-1:null) Could not find suitable Deployment Destination
> for
> > > this VM under any clusters, returning.
> > > 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) Searching all possible resources under this
> Zone: 1
> > > 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> > > that have (atleast one host with) enough CPU and RAM capacity under
> this
> > > Zone: 1
> > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) Removing from the clusterId list these clusters
> > from
> > > avoid set: [1]
> > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > (consoleproxy-1:null) No clusters found after removing disabled
> clusters
> > > and clusters in avoid list, returning.
> > > 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > (consoleproxy-1:null) VM state transitted from :Starting to Stopped
> with
> > > event: OperationFailedvm's original host id: null new host id: null
> host
> > > id before state transition: null
> > > 2013-10-29 09:22:52,045 WARN
> > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:null) Exception while trying to start console proxy
> > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create
> > > a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> > > com.cloud.dc.DataCenter; id=1
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > at
> > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > at
> > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> > > at
> > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> > > at
> > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> > > at
> > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> > > at
> > > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > at
> > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > >
> > >
> > > What am I missing? How can I get these two system VMs online?
> > >
> > > Best Regards,
> > >
> > >
> > >
> > > Adam Scarcella
> > >
> > >
> > > On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com>
> wrote:
> > >
> > > > It's worth a shot. I've gone through the db manually, and changed all
> > > > the places where I could see the old IPs, but your way is much more
> > > efficient.
> > > > Doing it now and will report back soon. Thanks.
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <
> > andrija.panic@gmail.com
> > > >wrote:
> > > >
> > > >> one of the crazy tries:
> > > >> backup your database, search for all IP addresses inside SQL file,
> > > >> map old to new IP addresses (old1 --> new1, old2 -->new2...) and
> then
> > > >> do search/replace each old one with new one.
> > > >> This is for netowrking/ip addresses (guest, public,etc), same for
> NFS
> > > >> serves, same for NFS host, etc...
> > > >>
> > > >> Not sure if that is going to make you UP again, but...
> > > >>
> > > >>
> > > >> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
> > > >>
> > > >> > Hi All,
> > > >> >
> > > >> > I know what I'm about to ask may seem like crazy talk, but I have
> a
> > > >> > situation at work where our CloudStack "box" was moved to another
> > > >> network.
> > > >> > This single server is a self-contained cloud with everything on
> one
> > > box.
> > > >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
> > > >> > primary
> > > >> and
> > > >> > secondary storage mounts. It's a very basic configuration, but my
> > > >> company
> > > >> > moved from one town to another, and the IT dept decided to move
> our
> > > >> server
> > > >> > to a different switch and subnet without asking. So now literally
> > > >> > every single IP address of any importance has been changed.
> > > >> >
> > > >> > I've been trying in vain to get this CS up and running again, but
> > > >> > it
> > > >> seems
> > > >> > like everything has changed including the guids, or at least that
> > > >> > seems
> > > >> to
> > > >> > be the case since I cannot match up guids from the database to
> > > >> > guids on
> > > >> the
> > > >> > primary/secondary NFS.
> > > >> >
> > > >> > I'm sure if each piece were separate and moved one piece at a
> time,
> > > >> > it would be fairly straightforward, but since this box is
> > > >> > self-contained
> > > >> and
> > > >> > everything changed at the same time, it is proving to be very
> > > difficult.
> > > >> >
> > > >> > Does anyone have any suggestions for me to attempt to recover my
> CS?
> > > >> >
> > > >> > Best Regards,
> > > >> >
> > > >> >
> > > >> >
> > > >> > Adam Scarcella
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Andrija Panić
> > > >> --------------------------------------
> > > >> http://admintweets.com
> > > >> --------------------------------------
> > > >>
> > > >
> > > >
> > > This email and any attachments to it may be confidential and are
> intended
> > > solely for the use of the individual to whom it is addressed. Any views
> > or
> > > opinions expressed are solely those of the author and do not
> necessarily
> > > represent those of Shape Blue Ltd or related companies. If you are not
> > the
> > > intended recipient of this email, you must neither take any action
> based
> > > upon its contents, nor copy or show it to anyone. Please contact the
> > sender
> > > if you believe you have received this email in error. Shape Blue Ltd
> is a
> > > company incorporated in England & Wales. ShapeBlue Services India LLP
> is
> > a
> > > company incorporated in India and is operated under license from Shape
> > Blue
> > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > Brasil
> > > and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > > registered trademark.
> > >
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Andrija Panic <an...@gmail.com>.
I suppose, you can recheck the network, and also check the AGENT
configuration file, make changes
there: /etc/cloudstack/agent/agent.properties
>From logs:
InsufficientServerCapacityException: Unable to create
> a deployment for VM.....
I had this kind of error, when my storage pools were disabled once, and
also when I have reached "disable capacity treshold" - my storage pool was
over 85% full.
So check if you secondaru storge is in maintance mode, or similar...
Also, destroying SSVM will force CS into making new one...
On 29 October 2013 16:09, Adam <ad...@gmail.com> wrote:
> Hi Geoff,
>
> Thanks for the reply, I was able to manually mount the secondary storage
> NFS:
>
> [root@cs-east-prod1 secondaryStorage]# hostname
> cs-east-prod1
> [root@cs-east-prod1 secondaryStorage]# hostname --fqdn
> cs-east-prod1.na.tibco.com
> [root@cs-east-prod1 secondaryStorage]#
> [root@cs-east-prod1 ~]# mkdir /secondaryStorage
> [root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
> /secondaryStorage
> [root@cs-east-prod1 ~]# cd /
> [root@cs-east-prod1 /]# ls -l
> total 120
> dr-xr-xr-x. 2 root root 4096 Oct 23 04:48 bin
> dr-xr-xr-x. 5 root root 3072 Oct 22 21:10 boot
> drwxr-xr-x. 10 root root 4096 Jun 25 10:23 cgroup
> drwxr-xr-x. 19 root root 3960 Oct 29 10:38 dev
> drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
> drwxrwxrwx. 4 root root 4096 Jul 3 20:10 export
> drwxr-xr-x. 3 root root 4096 Jul 3 19:14 home
> lrwxrwxrwx. 1 root root 25 Aug 28 15:46 Installers ->
> /var/www/html/Installers/
> lrwxrwxrwx. 1 root root 19 Jul 5 20:58 ISOs -> /var/www/html/ISOs/
> dr-xr-xr-x. 11 root root 4096 Jul 3 19:02 lib
> dr-xr-xr-x. 10 root root 12288 Oct 28 03:33 lib64
> drwx------. 2 root root 16384 Jul 3 18:44 lost+found
> drwxr-xr-x. 3 root root 4096 Oct 27 12:05 media
> drwxr-xr-x. 2 root root 0 Oct 29 10:08 misc
> drwxr-xr-x. 39 root root 4096 Sep 6 12:07 mnt
> drwxr-xr-x. 2 root root 0 Oct 29 10:08 net
> drwxr-xr-x. 3 root root 4096 Jul 3 19:05 opt
> drwxr-xr-x. 2 root root 4096 Oct 27 10:06 primarymount
> dr-xr-xr-x. 557 root root 0 Oct 29 10:08 proc
> dr-xr-x---. 35 root root 4096 Oct 29 10:09 root
> dr-xr-xr-x. 2 root root 12288 Oct 28 03:33 sbin
> drwxr-xr-x. 2 root root 4096 Oct 27 10:07 secondarymount
> drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondaryStorage
> drwxr-xr-x. 7 root root 0 Oct 29 10:08 selinux
> drwxr-xr-x. 2 root root 4096 Sep 23 2011 srv
> drwxr-xr-x. 13 root root 0 Oct 29 10:08 sys
> drwxrwxrwt. 16 root root 4096 Oct 29 11:07 tmp
> drwxr-xr-x. 13 root root 4096 Jul 3 18:55 usr
> drwxr-xr-x. 23 root root 4096 Jul 3 19:45 var
> [root@cs-east-prod1 /]# cd secondaryStorage/
> [root@cs-east-prod1 secondaryStorage]# ls -l
> total 12
> drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
> drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
> drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
> [root@cs-east-prod1 secondaryStorage]#
>
>
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
> geoff.higginbottom@shapeblue.com> wrote:
>
> > Adam,
> >
> > System VMs looping and failing to create is often caused by the Host
> being
> > unable to access Sec Storage.
> >
> > Do a manual mount of the Sec Storage NFS on the Host to confirm it can
> > access it.
> >
> > Regards
> >
> > Geoff Higginbottom
> >
> > D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
> >
> > geoff.higginbottom@shapeblue.com
> >
> > -----Original Message-----
> > From: Adam [mailto:adam.scarcella@gmail.com]
> > Sent: 29 October 2013 14:44
> > To: users@cloudstack.apache.org
> > Subject: Re: How to recover entire self contained CloudStack after major
> > network changes?
> >
> > I exported the db as a *.sql file, did search/replace and that seemed to
> > help a lot, but still not 100% out of the woods. Now I'm hitting the
> issue
> > where the System VMs (SecondaryStorageVm & ConsoleProxy) do not start.
> They
> > just keep looping from creation to being expunged.
> >
> > My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> > Primary Storage is online:
> >
> > [root@cs-east-prod1 ~]# df -kh
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/mapper/vg_cloudstackuse-lv_root
> > 5.7T 921G 4.5T 17% /
> > tmpfs 64G 72K 64G 1% /dev/shm
> > /dev/sda1 485M 118M 342M 26% /boot
> > 10.97.38.10:/export/primary
> > 5.7T 921G 4.5T 17%
> > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> >
> > Here is an excerpt from my most recent log:
> >
> > 2013-10-29 09:22:23,389 DEBUG
> > [network.router.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > 2013-10-29 09:22:23,391 DEBUG
> > [network.router.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > 2013-10-29 09:22:23,441 DEBUG
> > [network.router.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > 2013-10-29 09:22:23,443 DEBUG
> > [network.router.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> > (AgentManager-Handler-5:null) Ping from 10
> > 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-2:null) VmStatsCollector is running...
> > 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-3:null) StorageCollector is running...
> > 2013-10-29 09:22:38,716 INFO [storage.endpoint.DefaultEndPointSelector]
> > (StatsCollector-3:null) No running ssvm is found, so command will be sent
> > to LocalHostEndPoint
> > 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> > (StatsCollector-1:null) HostStatsCollector is running...
> > 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> > (StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
> > 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2013-10-29 09:22:51,802 DEBUG
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone
> 1
> > is ready to launch secondary storage VM
> > 2013-10-29 09:22:51,807 INFO
> > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> (secstorage-1:null)
> > No running secondary storage vms found in datacenter id=1, starting one
> > 2013-10-29 09:22:51,814 INFO
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No
> > stopped secondary storage vm is available, need to allocate a new
> secondary
> > storage vm
> > 2013-10-29 09:22:51,817 DEBUG
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Assign
> > secondary storage vm from a newly started instance for request from data
> > center : 1
> > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Found existing network configuration for offering
> > [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> > Ntwk[204|Guest|6]
> > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Releasing lock for
> > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Found existing network configuration for offering
> > [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
> > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Releasing lock for
> > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Found existing network configuration for offering
> > [Network Offering [2-Management-System-Management-Network]:
> > Ntwk[201|Management|2]
> > 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Releasing lock for
> > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Found existing network configuration for offering
> > [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
> > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Releasing lock for
> > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Allocating entries for VM:
> > VM[SecondaryStorageVm|s-4930-VM]
> > 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-4930-VM]
> > 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Allocating nic for vm
> VM[SecondaryStorageVm|s-4930-VM]
> > in network Ntwk[204|Guest|6] with requested profile
> > NicProfile[0-0-null-null-null
> > 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Allocating nic for vm
> VM[SecondaryStorageVm|s-4930-VM]
> > in network Ntwk[202|Control|3] with requested profile null
> > 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Allocating nic for vm
> VM[SecondaryStorageVm|s-4930-VM]
> > in network Ntwk[201|Management|2] with requested profile null
> > 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> > (secstorage-1:null) Allocating nic for vm
> VM[SecondaryStorageVm|s-4930-VM]
> > in network Ntwk[203|Storage|4] with requested profile null
> > 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-4930-VM]
> > 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Allocation completed for VM:
> > VM[SecondaryStorageVm|s-4930-VM]
> > 2013-10-29 09:22:51,925 DEBUG
> [cloud.alert.SecondaryStorageVmAlertAdapter]
> > (secstorage-1:null) received secondary storage vm alert
> > 2013-10-29 09:22:51,925 DEBUG
> [cloud.alert.SecondaryStorageVmAlertAdapter]
> > (secstorage-1:null) New secondary storage vm created, zone:
> EastProdZone1,
> > secStorageVm: s-4930-VM, public IP: null, private IP: null
> > 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> > (secstorage-1:null) VM state transitted from :Stopped to Starting with
> > event: StartRequestedvm's original host id: null new host id: null host
> id
> > before state transition: null
> > 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Successfully transitioned to start state for
> > VM[SecondaryStorageVm|s-4930-VM] reservation id =
> > 3b2435a0-9e32-4777-a42f-a766a375bc1b
> > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
> > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> > 2013-10-29 09:22:51,945 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> > 2013-10-29 09:22:51,946 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) DeploymentPlanner allocation algorithm:
> > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > 2013-10-29 09:22:51,946 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) Trying to allocate a host and storage pools from
> dc:1,
> > pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> > 2013-10-29 09:22:51,946 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
> > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) Searching all possible resources under this Zone: 1
> > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) Listing clusters in order of aggregate capacity, that
> > have (atleast one host with) enough CPU and RAM capacity under this
> Zone: 1
> > 2013-10-29 09:22:51,958 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> > 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> > (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
> > cluster:1
> > 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> > (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to
> > check for allocation: []
> > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> after
> > prioritization: []
> > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=256
> > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > suitable hosts
> > 2013-10-29 09:22:51,965 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) No suitable hosts found
> > 2013-10-29 09:22:51,965 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) No suitable hosts found under this Cluster: 1
> > 2013-10-29 09:22:51,968 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (secstorage-1:null) Could not find suitable Deployment Destination for
> > this VM under any clusters, returning.
> > 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) Searching all possible resources under this Zone: 1
> > 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) Listing clusters in order of aggregate capacity, that
> > have (atleast one host with) enough CPU and RAM capacity under this
> Zone: 1
> > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) Removing from the clusterId list these clusters from
> > avoid set: [1]
> > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > (secstorage-1:null) No clusters found after removing disabled clusters
> and
> > clusters in avoid list, returning.
> > 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> > (secstorage-1:null) VM state transitted from :Starting to Stopped with
> > event: OperationFailedvm's original host id: null new host id: null host
> > id before state transition: null
> > 2013-10-29 09:22:51,985 WARN
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Exception while trying to start secondary storage vm
> > com.cloud.exception.InsufficientServerCapacityException: Unable to create
> > a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=1
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > at
> >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> > at
> >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> > at
> >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> > at
> >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > at
> >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> > at
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > 2013-10-29 09:22:51,987 INFO
> > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Unable to start secondary storage vm for standby capacity, secStorageVm
> vm
> > Id :
> > 4930, will recycle it and start a new one
> > 2013-10-29 09:22:51,987 INFO
> > [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> (secstorage-1:null)
> > Primary secondary storage is not even started, wait until next turn
> > 2013-10-29 09:22:52,009 DEBUG
> [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> > 2013-10-29 09:22:52,009 DEBUG
> [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > (consoleproxy-1:null) Expand console proxy standby capacity for zone
> > EastProdZone1
> > 2013-10-29 09:22:52,012 INFO
> [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> > running pool. proxy vm id : 4271
> > 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> > (consoleproxy-1:null) VM state transitted from :Stopped to Starting with
> > event: StartRequestedvm's original host id: null new host id: null host
> id
> > before state transition: null
> > 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (consoleproxy-1:null) Successfully transitioned to start state for
> > VM[ConsoleProxy|v-4271-VM] reservation id =
> > 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> > 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
> > 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts:
> null
> > 2013-10-29 09:22:52,023 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts:
> null
> > 2013-10-29 09:22:52,024 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> > com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > 2013-10-29 09:22:52,024 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) Trying to allocate a host and storage pools from
> > dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
> 1073741824
> > 2013-10-29 09:22:52,024 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
> > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) Searching all possible resources under this Zone: 1
> > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> > that have (atleast one host with) enough CPU and RAM capacity under this
> > Zone: 1
> > 2013-10-29 09:22:52,031 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> > 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> > (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> pod:1
> > cluster:1
> > 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> > (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts
> to
> > check for allocation: []
> > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> > after prioritization: []
> > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > Ram=1024
> > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > suitable hosts
> > 2013-10-29 09:22:52,034 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) No suitable hosts found
> > 2013-10-29 09:22:52,034 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> > 2013-10-29 09:22:52,036 DEBUG
> [cloud.deploy.DeploymentPlanningManagerImpl]
> > (consoleproxy-1:null) Could not find suitable Deployment Destination for
> > this VM under any clusters, returning.
> > 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) Searching all possible resources under this Zone: 1
> > 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> > that have (atleast one host with) enough CPU and RAM capacity under this
> > Zone: 1
> > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) Removing from the clusterId list these clusters
> from
> > avoid set: [1]
> > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > (consoleproxy-1:null) No clusters found after removing disabled clusters
> > and clusters in avoid list, returning.
> > 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> > (consoleproxy-1:null) VM state transitted from :Starting to Stopped with
> > event: OperationFailedvm's original host id: null new host id: null host
> > id before state transition: null
> > 2013-10-29 09:22:52,045 WARN
> [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > (consoleproxy-1:null) Exception while trying to start console proxy
> > com.cloud.exception.InsufficientServerCapacityException: Unable to create
> > a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=1
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > at
> >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > at
> >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> > at
> >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> > at
> >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> > at
> >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> > at
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> >
> >
> > What am I missing? How can I get these two system VMs online?
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com> wrote:
> >
> > > It's worth a shot. I've gone through the db manually, and changed all
> > > the places where I could see the old IPs, but your way is much more
> > efficient.
> > > Doing it now and will report back soon. Thanks.
> > >
> > > Best Regards,
> > >
> > >
> > >
> > > Adam Scarcella
> > >
> > >
> > > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <
> andrija.panic@gmail.com
> > >wrote:
> > >
> > >> one of the crazy tries:
> > >> backup your database, search for all IP addresses inside SQL file,
> > >> map old to new IP addresses (old1 --> new1, old2 -->new2...) and then
> > >> do search/replace each old one with new one.
> > >> This is for netowrking/ip addresses (guest, public,etc), same for NFS
> > >> serves, same for NFS host, etc...
> > >>
> > >> Not sure if that is going to make you UP again, but...
> > >>
> > >>
> > >> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > I know what I'm about to ask may seem like crazy talk, but I have a
> > >> > situation at work where our CloudStack "box" was moved to another
> > >> network.
> > >> > This single server is a self-contained cloud with everything on one
> > box.
> > >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
> > >> > primary
> > >> and
> > >> > secondary storage mounts. It's a very basic configuration, but my
> > >> company
> > >> > moved from one town to another, and the IT dept decided to move our
> > >> server
> > >> > to a different switch and subnet without asking. So now literally
> > >> > every single IP address of any importance has been changed.
> > >> >
> > >> > I've been trying in vain to get this CS up and running again, but
> > >> > it
> > >> seems
> > >> > like everything has changed including the guids, or at least that
> > >> > seems
> > >> to
> > >> > be the case since I cannot match up guids from the database to
> > >> > guids on
> > >> the
> > >> > primary/secondary NFS.
> > >> >
> > >> > I'm sure if each piece were separate and moved one piece at a time,
> > >> > it would be fairly straightforward, but since this box is
> > >> > self-contained
> > >> and
> > >> > everything changed at the same time, it is proving to be very
> > difficult.
> > >> >
> > >> > Does anyone have any suggestions for me to attempt to recover my CS?
> > >> >
> > >> > Best Regards,
> > >> >
> > >> >
> > >> >
> > >> > Adam Scarcella
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Andrija Panić
> > >> --------------------------------------
> > >> http://admintweets.com
> > >> --------------------------------------
> > >>
> > >
> > >
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender
> > if you believe you have received this email in error. Shape Blue Ltd is a
> > company incorporated in England & Wales. ShapeBlue Services India LLP is
> a
> > company incorporated in India and is operated under license from Shape
> Blue
> > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil
> > and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > registered trademark.
> >
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Adam <ad...@gmail.com>.
Hi Geoff,
Thanks for the reply, I was able to manually mount the secondary storage
NFS:
[root@cs-east-prod1 secondaryStorage]# hostname
cs-east-prod1
[root@cs-east-prod1 secondaryStorage]# hostname --fqdn
cs-east-prod1.na.tibco.com
[root@cs-east-prod1 secondaryStorage]#
[root@cs-east-prod1 ~]# mkdir /secondaryStorage
[root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
/secondaryStorage
[root@cs-east-prod1 ~]# cd /
[root@cs-east-prod1 /]# ls -l
total 120
dr-xr-xr-x. 2 root root 4096 Oct 23 04:48 bin
dr-xr-xr-x. 5 root root 3072 Oct 22 21:10 boot
drwxr-xr-x. 10 root root 4096 Jun 25 10:23 cgroup
drwxr-xr-x. 19 root root 3960 Oct 29 10:38 dev
drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
drwxrwxrwx. 4 root root 4096 Jul 3 20:10 export
drwxr-xr-x. 3 root root 4096 Jul 3 19:14 home
lrwxrwxrwx. 1 root root 25 Aug 28 15:46 Installers ->
/var/www/html/Installers/
lrwxrwxrwx. 1 root root 19 Jul 5 20:58 ISOs -> /var/www/html/ISOs/
dr-xr-xr-x. 11 root root 4096 Jul 3 19:02 lib
dr-xr-xr-x. 10 root root 12288 Oct 28 03:33 lib64
drwx------. 2 root root 16384 Jul 3 18:44 lost+found
drwxr-xr-x. 3 root root 4096 Oct 27 12:05 media
drwxr-xr-x. 2 root root 0 Oct 29 10:08 misc
drwxr-xr-x. 39 root root 4096 Sep 6 12:07 mnt
drwxr-xr-x. 2 root root 0 Oct 29 10:08 net
drwxr-xr-x. 3 root root 4096 Jul 3 19:05 opt
drwxr-xr-x. 2 root root 4096 Oct 27 10:06 primarymount
dr-xr-xr-x. 557 root root 0 Oct 29 10:08 proc
dr-xr-x---. 35 root root 4096 Oct 29 10:09 root
dr-xr-xr-x. 2 root root 12288 Oct 28 03:33 sbin
drwxr-xr-x. 2 root root 4096 Oct 27 10:07 secondarymount
drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondaryStorage
drwxr-xr-x. 7 root root 0 Oct 29 10:08 selinux
drwxr-xr-x. 2 root root 4096 Sep 23 2011 srv
drwxr-xr-x. 13 root root 0 Oct 29 10:08 sys
drwxrwxrwt. 16 root root 4096 Oct 29 11:07 tmp
drwxr-xr-x. 13 root root 4096 Jul 3 18:55 usr
drwxr-xr-x. 23 root root 4096 Jul 3 19:45 var
[root@cs-east-prod1 /]# cd secondaryStorage/
[root@cs-east-prod1 secondaryStorage]# ls -l
total 12
drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
[root@cs-east-prod1 secondaryStorage]#
Best Regards,
Adam Scarcella
On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
geoff.higginbottom@shapeblue.com> wrote:
> Adam,
>
> System VMs looping and failing to create is often caused by the Host being
> unable to access Sec Storage.
>
> Do a manual mount of the Sec Storage NFS on the Host to confirm it can
> access it.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbottom@shapeblue.com
>
> -----Original Message-----
> From: Adam [mailto:adam.scarcella@gmail.com]
> Sent: 29 October 2013 14:44
> To: users@cloudstack.apache.org
> Subject: Re: How to recover entire self contained CloudStack after major
> network changes?
>
> I exported the db as a *.sql file, did search/replace and that seemed to
> help a lot, but still not 100% out of the woods. Now I'm hitting the issue
> where the System VMs (SecondaryStorageVm & ConsoleProxy) do not start. They
> just keep looping from creation to being expunged.
>
> My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> Primary Storage is online:
>
> [root@cs-east-prod1 ~]# df -kh
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/vg_cloudstackuse-lv_root
> 5.7T 921G 4.5T 17% /
> tmpfs 64G 72K 64G 1% /dev/shm
> /dev/sda1 485M 118M 342M 26% /boot
> 10.97.38.10:/export/primary
> 5.7T 921G 4.5T 17%
> /mnt/910365e6-a298-37b8-904f-2b49198c8286
>
> Here is an excerpt from my most recent log:
>
> 2013-10-29 09:22:23,389 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers to update status.
> 2013-10-29 09:22:23,391 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-10-29 09:22:23,441 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers to update status.
> 2013-10-29 09:22:23,443 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-5:null) Ping from 10
> 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) VmStatsCollector is running...
> 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-3:null) StorageCollector is running...
> 2013-10-29 09:22:38,716 INFO [storage.endpoint.DefaultEndPointSelector]
> (StatsCollector-3:null) No running ssvm is found, so command will be sent
> to LocalHostEndPoint
> 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) HostStatsCollector is running...
> 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
> 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-10-29 09:22:51,802 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1
> is ready to launch secondary storage VM
> 2013-10-29 09:22:51,807 INFO
> [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
> No running secondary storage vms found in datacenter id=1, starting one
> 2013-10-29 09:22:51,814 INFO
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No
> stopped secondary storage vm is available, need to allocate a new secondary
> storage vm
> 2013-10-29 09:22:51,817 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Assign
> secondary storage vm from a newly started instance for request from data
> center : 1
> 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Found existing network configuration for offering
> [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> Ntwk[204|Guest|6]
> 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Releasing lock for
> Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Found existing network configuration for offering
> [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
> 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Releasing lock for
> Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Found existing network configuration for offering
> [Network Offering [2-Management-System-Management-Network]:
> Ntwk[201|Management|2]
> 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Releasing lock for
> Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Found existing network configuration for offering
> [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
> 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Releasing lock for
> Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Allocating entries for VM:
> VM[SecondaryStorageVm|s-4930-VM]
> 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-4930-VM]
> 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
> in network Ntwk[204|Guest|6] with requested profile
> NicProfile[0-0-null-null-null
> 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
> in network Ntwk[202|Control|3] with requested profile null
> 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
> in network Ntwk[201|Management|2] with requested profile null
> 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
> in network Ntwk[203|Storage|4] with requested profile null
> 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-4930-VM]
> 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Allocation completed for VM:
> VM[SecondaryStorageVm|s-4930-VM]
> 2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
> (secstorage-1:null) received secondary storage vm alert
> 2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
> (secstorage-1:null) New secondary storage vm created, zone: EastProdZone1,
> secStorageVm: s-4930-VM, public IP: null, private IP: null
> 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> (secstorage-1:null) VM state transitted from :Stopped to Starting with
> event: StartRequestedvm's original host id: null new host id: null host id
> before state transition: null
> 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Successfully transitioned to start state for
> VM[SecondaryStorageVm|s-4930-VM] reservation id =
> 3b2435a0-9e32-4777-a42f-a766a375bc1b
> 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
> 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> 2013-10-29 09:22:51,945 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) DeploymentPlanner allocation algorithm:
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
> 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) Searching all possible resources under this Zone: 1
> 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) Listing clusters in order of aggregate capacity, that
> have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
> 2013-10-29 09:22:51,958 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
> cluster:1
> 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to
> check for allocation: []
> 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation after
> prioritization: []
> 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=256
> 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> suitable hosts
> 2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) No suitable hosts found
> 2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) No suitable hosts found under this Cluster: 1
> 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (secstorage-1:null) Could not find suitable Deployment Destination for
> this VM under any clusters, returning.
> 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) Searching all possible resources under this Zone: 1
> 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) Listing clusters in order of aggregate capacity, that
> have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
> 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) Removing from the clusterId list these clusters from
> avoid set: [1]
> 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> (secstorage-1:null) No clusters found after removing disabled clusters and
> clusters in avoid list, returning.
> 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> (secstorage-1:null) VM state transitted from :Starting to Stopped with
> event: OperationFailedvm's original host id: null new host id: null host
> id before state transition: null
> 2013-10-29 09:22:51,985 WARN
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Exception while trying to start secondary storage vm
> com.cloud.exception.InsufficientServerCapacityException: Unable to create
> a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> com.cloud.dc.DataCenter; id=1
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> at
>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> at
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at
>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at
> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2013-10-29 09:22:51,987 INFO
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> Unable to start secondary storage vm for standby capacity, secStorageVm vm
> Id :
> 4930, will recycle it and start a new one
> 2013-10-29 09:22:51,987 INFO
> [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
> Primary secondary storage is not even started, wait until next turn
> 2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> 2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Expand console proxy standby capacity for zone
> EastProdZone1
> 2013-10-29 09:22:52,012 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> running pool. proxy vm id : 4271
> 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> (consoleproxy-1:null) VM state transitted from :Stopped to Starting with
> event: StartRequestedvm's original host id: null new host id: null host id
> before state transition: null
> 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Successfully transitioned to start state for
> VM[ConsoleProxy|v-4271-VM] reservation id =
> 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
> 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> 2013-10-29 09:22:52,023 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
> 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) Trying to allocate a host and storage pools from
> dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 1073741824
> 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
> 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) Searching all possible resources under this Zone: 1
> 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> that have (atleast one host with) enough CPU and RAM capacity under this
> Zone: 1
> 2013-10-29 09:22:52,031 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
> cluster:1
> 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to
> check for allocation: []
> 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> after prioritization: []
> 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> Ram=1024
> 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
> suitable hosts
> 2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) No suitable hosts found
> 2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (consoleproxy-1:null) Could not find suitable Deployment Destination for
> this VM under any clusters, returning.
> 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) Searching all possible resources under this Zone: 1
> 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) Listing clusters in order of aggregate capacity,
> that have (atleast one host with) enough CPU and RAM capacity under this
> Zone: 1
> 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) Removing from the clusterId list these clusters from
> avoid set: [1]
> 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> (consoleproxy-1:null) No clusters found after removing disabled clusters
> and clusters in avoid list, returning.
> 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> (consoleproxy-1:null) VM state transitted from :Starting to Stopped with
> event: OperationFailedvm's original host id: null new host id: null host
> id before state transition: null
> 2013-10-29 09:22:52,045 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Exception while trying to start console proxy
> com.cloud.exception.InsufficientServerCapacityException: Unable to create
> a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> com.cloud.dc.DataCenter; id=1
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> at
>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at
>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> at
>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> at
>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at
> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> What am I missing? How can I get these two system VMs online?
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com> wrote:
>
> > It's worth a shot. I've gone through the db manually, and changed all
> > the places where I could see the old IPs, but your way is much more
> efficient.
> > Doing it now and will report back soon. Thanks.
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <andrija.panic@gmail.com
> >wrote:
> >
> >> one of the crazy tries:
> >> backup your database, search for all IP addresses inside SQL file,
> >> map old to new IP addresses (old1 --> new1, old2 -->new2...) and then
> >> do search/replace each old one with new one.
> >> This is for netowrking/ip addresses (guest, public,etc), same for NFS
> >> serves, same for NFS host, etc...
> >>
> >> Not sure if that is going to make you UP again, but...
> >>
> >>
> >> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
> >>
> >> > Hi All,
> >> >
> >> > I know what I'm about to ask may seem like crazy talk, but I have a
> >> > situation at work where our CloudStack "box" was moved to another
> >> network.
> >> > This single server is a self-contained cloud with everything on one
> box.
> >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
> >> > primary
> >> and
> >> > secondary storage mounts. It's a very basic configuration, but my
> >> company
> >> > moved from one town to another, and the IT dept decided to move our
> >> server
> >> > to a different switch and subnet without asking. So now literally
> >> > every single IP address of any importance has been changed.
> >> >
> >> > I've been trying in vain to get this CS up and running again, but
> >> > it
> >> seems
> >> > like everything has changed including the guids, or at least that
> >> > seems
> >> to
> >> > be the case since I cannot match up guids from the database to
> >> > guids on
> >> the
> >> > primary/secondary NFS.
> >> >
> >> > I'm sure if each piece were separate and moved one piece at a time,
> >> > it would be fairly straightforward, but since this box is
> >> > self-contained
> >> and
> >> > everything changed at the same time, it is proving to be very
> difficult.
> >> >
> >> > Does anyone have any suggestions for me to attempt to recover my CS?
> >> >
> >> > Best Regards,
> >> >
> >> >
> >> >
> >> > Adam Scarcella
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >> --------------------------------------
> >> http://admintweets.com
> >> --------------------------------------
> >>
> >
> >
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.
>
RE: How to recover entire self contained CloudStack after major
network changes?
Posted by Geoff Higginbottom <ge...@shapeblue.com>.
Adam,
System VMs looping and failing to create is often caused by the Host being unable to access Sec Storage.
Do a manual mount of the Sec Storage NFS on the Host to confirm it can access it.
Regards
Geoff Higginbottom
D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
geoff.higginbottom@shapeblue.com
-----Original Message-----
From: Adam [mailto:adam.scarcella@gmail.com]
Sent: 29 October 2013 14:44
To: users@cloudstack.apache.org
Subject: Re: How to recover entire self contained CloudStack after major network changes?
I exported the db as a *.sql file, did search/replace and that seemed to help a lot, but still not 100% out of the woods. Now I'm hitting the issue where the System VMs (SecondaryStorageVm & ConsoleProxy) do not start. They just keep looping from creation to being expunged.
My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up, Primary Storage is online:
[root@cs-east-prod1 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_cloudstackuse-lv_root
5.7T 921G 4.5T 17% /
tmpfs 64G 72K 64G 1% /dev/shm
/dev/sda1 485M 118M 342M 26% /boot
10.97.38.10:/export/primary
5.7T 921G 4.5T 17%
/mnt/910365e6-a298-37b8-904f-2b49198c8286
Here is an excerpt from my most recent log:
2013-10-29 09:22:23,389 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2013-10-29 09:22:23,391 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-10-29 09:22:23,441 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2013-10-29 09:22:23,443 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) Ping from 10
2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) VmStatsCollector is running...
2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
(StatsCollector-3:null) StorageCollector is running...
2013-10-29 09:22:38,716 INFO [storage.endpoint.DefaultEndPointSelector]
(StatsCollector-3:null) No running ssvm is found, so command will be sent to LocalHostEndPoint
2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) HostStatsCollector is running...
2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2013-10-29 09:22:51,802 DEBUG
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2013-10-29 09:22:51,807 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null) No running secondary storage vms found in datacenter id=1, starting one
2013-10-29 09:22:51,814 INFO
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
2013-10-29 09:22:51,817 DEBUG
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Assign secondary storage vm from a newly started instance for request from data center : 1
2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
Ntwk[204|Guest|6]
2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]:
Ntwk[201|Management|2]
2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating entries for VM:
VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM] in network Ntwk[204|Guest|6] with requested profile NicProfile[0-0-null-null-null
2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM] in network Ntwk[202|Control|3] with requested profile null
2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM] in network Ntwk[201|Management|2] with requested profile null
2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM] in network Ntwk[203|Storage|4] with requested profile null
2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocation completed for VM:
VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) received secondary storage vm alert
2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) New secondary storage vm created, zone: EastProdZone1,
secStorageVm: s-4930-VM, public IP: null, private IP: null
2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Stopped to Starting with
event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Successfully transitioned to start state for VM[SecondaryStorageVm|s-4930-VM] reservation id = 3b2435a0-9e32-4777-a42f-a766a375bc1b
2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:51,945 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:51,958 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
cluster:1
2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to check for allocation: []
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation after
prioritization: []
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=256
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) No suitable hosts found
2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) No suitable hosts found under this Cluster: 1
2013-10-29 09:22:51,968 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Removing from the clusterId list these clusters from avoid set: [1]
2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id before state transition: null
2013-10-29 09:22:51,985 WARN
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2013-10-29 09:22:51,987 INFO
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id :
4930, will recycle it and start a new one
2013-10-29 09:22:51,987 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null) Primary secondary storage is not even started, wait until next turn
2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is ready to launch console proxy
2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Expand console proxy standby capacity for zone
EastProdZone1
2013-10-29 09:22:52,012 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Found a stopped console proxy, bring it up to running pool. proxy vm id : 4271
2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Stopped to Starting with
event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Successfully transitioned to start state for VM[ConsoleProxy|v-4271-VM] reservation id =
11861d83-fc09-4b25-8c07-bcb6ad8e7f54
2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:52,023 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 1073741824
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:52,031 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
cluster:1
2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to check for allocation: []
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for allocation after prioritization: []
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=1024
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) No suitable hosts found
2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) No suitable hosts found under this Cluster: 1
2013-10-29 09:22:52,036 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Removing from the clusterId list these clusters from avoid set: [1]
2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id before state transition: null
2013-10-29 09:22:52,045 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
What am I missing? How can I get these two system VMs online?
Best Regards,
Adam Scarcella
On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com> wrote:
> It's worth a shot. I've gone through the db manually, and changed all
> the places where I could see the old IPs, but your way is much more efficient.
> Doing it now and will report back soon. Thanks.
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <an...@gmail.com>wrote:
>
>> one of the crazy tries:
>> backup your database, search for all IP addresses inside SQL file,
>> map old to new IP addresses (old1 --> new1, old2 -->new2...) and then
>> do search/replace each old one with new one.
>> This is for netowrking/ip addresses (guest, public,etc), same for NFS
>> serves, same for NFS host, etc...
>>
>> Not sure if that is going to make you UP again, but...
>>
>>
>> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
>>
>> > Hi All,
>> >
>> > I know what I'm about to ask may seem like crazy talk, but I have a
>> > situation at work where our CloudStack "box" was moved to another
>> network.
>> > This single server is a self-contained cloud with everything on one box.
>> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for
>> > primary
>> and
>> > secondary storage mounts. It's a very basic configuration, but my
>> company
>> > moved from one town to another, and the IT dept decided to move our
>> server
>> > to a different switch and subnet without asking. So now literally
>> > every single IP address of any importance has been changed.
>> >
>> > I've been trying in vain to get this CS up and running again, but
>> > it
>> seems
>> > like everything has changed including the guids, or at least that
>> > seems
>> to
>> > be the case since I cannot match up guids from the database to
>> > guids on
>> the
>> > primary/secondary NFS.
>> >
>> > I'm sure if each piece were separate and moved one piece at a time,
>> > it would be fairly straightforward, but since this box is
>> > self-contained
>> and
>> > everything changed at the same time, it is proving to be very difficult.
>> >
>> > Does anyone have any suggestions for me to attempt to recover my CS?
>> >
>> > Best Regards,
>> >
>> >
>> >
>> > Adam Scarcella
>> >
>>
>>
>>
>> --
>>
>> Andrija Panić
>> --------------------------------------
>> http://admintweets.com
>> --------------------------------------
>>
>
>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Adam <ad...@gmail.com>.
I exported the db as a *.sql file, did search/replace and that seemed to
help a lot, but still not 100% out of the woods. Now I'm hitting the issue
where the System VMs (SecondaryStorageVm & ConsoleProxy) do not start. They
just keep looping from creation to being expunged.
My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up, Primary
Storage is online:
[root@cs-east-prod1 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_cloudstackuse-lv_root
5.7T 921G 4.5T 17% /
tmpfs 64G 72K 64G 1% /dev/shm
/dev/sda1 485M 118M 342M 26% /boot
10.97.38.10:/export/primary
5.7T 921G 4.5T 17%
/mnt/910365e6-a298-37b8-904f-2b49198c8286
Here is an excerpt from my most recent log:
2013-10-29 09:22:23,389 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2013-10-29 09:22:23,391 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-10-29 09:22:23,441 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2013-10-29 09:22:23,443 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) Ping from 10
2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) VmStatsCollector is running...
2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
(StatsCollector-3:null) StorageCollector is running...
2013-10-29 09:22:38,716 INFO [storage.endpoint.DefaultEndPointSelector]
(StatsCollector-3:null) No running ssvm is found, so command will be sent
to LocalHostEndPoint
2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) HostStatsCollector is running...
2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 10-768344083: Received: { Ans: , MgmtId:
86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2013-10-29 09:22:51,802 DEBUG
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1
is ready to launch secondary storage VM
2013-10-29 09:22:51,807 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
No running secondary storage vms found in datacenter id=1, starting one
2013-10-29 09:22:51,814 INFO
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
2013-10-29 09:22:51,817 DEBUG
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Assign
secondary storage vm from a newly started instance for request from data
center : 1
2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering
[Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
Ntwk[204|Guest|6]
2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering
[Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering
[Network Offering [2-Management-System-Management-Network]:
Ntwk[201|Management|2]
2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering
[Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating entries for VM:
VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
in network Ntwk[204|Guest|6] with requested profile
NicProfile[0-0-null-null-null
2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
in network Ntwk[202|Control|3] with requested profile null
2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
in network Ntwk[201|Management|2] with requested profile null
2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-4930-VM]
in network Ntwk[203|Storage|4] with requested profile null
2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocation completed for VM:
VM[SecondaryStorageVm|s-4930-VM]
2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) received secondary storage vm alert
2013-10-29 09:22:51,925 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) New secondary storage vm created, zone: EastProdZone1,
secStorageVm: s-4930-VM, public IP: null, private IP: null
2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Stopped to Starting with
event: StartRequestedvm's original host id: null new host id: null host id
before state transition: null
2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Successfully transitioned to start state for
VM[SecondaryStorageVm|s-4930-VM] reservation id =
3b2435a0-9e32-4777-a42f-a766a375bc1b
2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:51,945 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Listing clusters in order of aggregate capacity, that
have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:51,958 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
cluster:1
2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to
check for allocation: []
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation after
prioritization: []
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=256
2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
(secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0 suitable
hosts
2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) No suitable hosts found
2013-10-29 09:22:51,965 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) No suitable hosts found under this Cluster: 1
2013-10-29 09:22:51,968 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Could not find suitable Deployment Destination for this
VM under any clusters, returning.
2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Listing clusters in order of aggregate capacity, that
have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Removing from the clusterId list these clusters from
avoid set: [1]
2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) No clusters found after removing disabled clusters and
clusters in avoid list, returning.
2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id
before state transition: null
2013-10-29 09:22:51,985 WARN
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2013-10-29 09:22:51,987 INFO
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable
to start secondary storage vm for standby capacity, secStorageVm vm Id :
4930, will recycle it and start a new one
2013-10-29 09:22:51,987 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
Primary secondary storage is not even started, wait until next turn
2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is ready to launch console proxy
2013-10-29 09:22:52,009 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Expand console proxy standby capacity for zone
EastProdZone1
2013-10-29 09:22:52,012 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Found a stopped console proxy, bring it up to running
pool. proxy vm id : 4271
2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Stopped to Starting with
event: StartRequestedvm's original host id: null new host id: null host id
before state transition: null
2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Successfully transitioned to start state for
VM[ConsoleProxy|v-4271-VM] reservation id =
11861d83-fc09-4b25-8c07-bcb6ad8e7f54
2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:52,023 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Trying to allocate a host and storage pools from
dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 1073741824
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Listing clusters in order of aggregate capacity, that
have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:52,031 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1
cluster:1
2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts to
check for allocation: []
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
after prioritization: []
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=1024
2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
(consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
suitable hosts
2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) No suitable hosts found
2013-10-29 09:22:52,034 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) No suitable hosts found under this Cluster: 1
2013-10-29 09:22:52,036 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Could not find suitable Deployment Destination for
this VM under any clusters, returning.
2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Searching all possible resources under this Zone: 1
2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Listing clusters in order of aggregate capacity, that
have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Removing from the clusterId list these clusters from
avoid set: [1]
2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) No clusters found after removing disabled clusters
and clusters in avoid list, returning.
2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id
before state transition: null
2013-10-29 09:22:52,045 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
What am I missing? How can I get these two system VMs online?
Best Regards,
Adam Scarcella
On Mon, Oct 28, 2013 at 8:07 PM, Adam <ad...@gmail.com> wrote:
> It's worth a shot. I've gone through the db manually, and changed all the
> places where I could see the old IPs, but your way is much more efficient.
> Doing it now and will report back soon. Thanks.
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <an...@gmail.com>wrote:
>
>> one of the crazy tries:
>> backup your database, search for all IP addresses inside SQL file, map old
>> to new IP addresses (old1 --> new1, old2 -->new2...) and then do
>> search/replace each old one with new one.
>> This is for netowrking/ip addresses (guest, public,etc), same for NFS
>> serves, same for NFS host, etc...
>>
>> Not sure if that is going to make you UP again, but...
>>
>>
>> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
>>
>> > Hi All,
>> >
>> > I know what I'm about to ask may seem like crazy talk, but I have a
>> > situation at work where our CloudStack "box" was moved to another
>> network.
>> > This single server is a self-contained cloud with everything on one box.
>> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for primary
>> and
>> > secondary storage mounts. It's a very basic configuration, but my
>> company
>> > moved from one town to another, and the IT dept decided to move our
>> server
>> > to a different switch and subnet without asking. So now literally every
>> > single IP address of any importance has been changed.
>> >
>> > I've been trying in vain to get this CS up and running again, but it
>> seems
>> > like everything has changed including the guids, or at least that seems
>> to
>> > be the case since I cannot match up guids from the database to guids on
>> the
>> > primary/secondary NFS.
>> >
>> > I'm sure if each piece were separate and moved one piece at a time, it
>> > would be fairly straightforward, but since this box is self-contained
>> and
>> > everything changed at the same time, it is proving to be very difficult.
>> >
>> > Does anyone have any suggestions for me to attempt to recover my CS?
>> >
>> > Best Regards,
>> >
>> >
>> >
>> > Adam Scarcella
>> >
>>
>>
>>
>> --
>>
>> Andrija Panić
>> --------------------------------------
>> http://admintweets.com
>> --------------------------------------
>>
>
>
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Adam <ad...@gmail.com>.
It's worth a shot. I've gone through the db manually, and changed all the
places where I could see the old IPs, but your way is much more efficient.
Doing it now and will report back soon. Thanks.
Best Regards,
Adam Scarcella
On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <an...@gmail.com>wrote:
> one of the crazy tries:
> backup your database, search for all IP addresses inside SQL file, map old
> to new IP addresses (old1 --> new1, old2 -->new2...) and then do
> search/replace each old one with new one.
> This is for netowrking/ip addresses (guest, public,etc), same for NFS
> serves, same for NFS host, etc...
>
> Not sure if that is going to make you UP again, but...
>
>
> On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
>
> > Hi All,
> >
> > I know what I'm about to ask may seem like crazy talk, but I have a
> > situation at work where our CloudStack "box" was moved to another
> network.
> > This single server is a self-contained cloud with everything on one box.
> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for primary
> and
> > secondary storage mounts. It's a very basic configuration, but my company
> > moved from one town to another, and the IT dept decided to move our
> server
> > to a different switch and subnet without asking. So now literally every
> > single IP address of any importance has been changed.
> >
> > I've been trying in vain to get this CS up and running again, but it
> seems
> > like everything has changed including the guids, or at least that seems
> to
> > be the case since I cannot match up guids from the database to guids on
> the
> > primary/secondary NFS.
> >
> > I'm sure if each piece were separate and moved one piece at a time, it
> > would be fairly straightforward, but since this box is self-contained and
> > everything changed at the same time, it is proving to be very difficult.
> >
> > Does anyone have any suggestions for me to attempt to recover my CS?
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
> http://admintweets.com
> --------------------------------------
>
Re: How to recover entire self contained CloudStack after major
network changes?
Posted by Andrija Panic <an...@gmail.com>.
one of the crazy tries:
backup your database, search for all IP addresses inside SQL file, map old
to new IP addresses (old1 --> new1, old2 -->new2...) and then do
search/replace each old one with new one.
This is for netowrking/ip addresses (guest, public,etc), same for NFS
serves, same for NFS host, etc...
Not sure if that is going to make you UP again, but...
On 28 October 2013 22:59, Adam <ad...@gmail.com> wrote:
> Hi All,
>
> I know what I'm about to ask may seem like crazy talk, but I have a
> situation at work where our CloudStack "box" was moved to another network.
> This single server is a self-contained cloud with everything on one box.
> I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS for primary and
> secondary storage mounts. It's a very basic configuration, but my company
> moved from one town to another, and the IT dept decided to move our server
> to a different switch and subnet without asking. So now literally every
> single IP address of any importance has been changed.
>
> I've been trying in vain to get this CS up and running again, but it seems
> like everything has changed including the guids, or at least that seems to
> be the case since I cannot match up guids from the database to guids on the
> primary/secondary NFS.
>
> I'm sure if each piece were separate and moved one piece at a time, it
> would be fairly straightforward, but since this box is self-contained and
> everything changed at the same time, it is proving to be very difficult.
>
> Does anyone have any suggestions for me to attempt to recover my CS?
>
> Best Regards,
>
>
>
> Adam Scarcella
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------