You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Dave Cundiff <sy...@gmail.com> on 2012/11/16 21:41:49 UTC
System VMs not rebuilding
Hello,
It appears the 4.0 upgrade has brought with it some bugs! I tried to
rebuild my system VMs by erasing them via the web console. Now they just
don't build. All I'm getting in the logs are single lines stating they
don't exist or the zone is not ready to launch them. No attempts to build a
VM occur. The zone/pods/hosts are all enabled.
Output I'm getting in the logs.
2012-11-16 15:35:47,466 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) There is no secondary storage VM for secondary
storage host nfs://10.11.0.250/
2012-11-16 15:36:54,547 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
Performing an operation that requires the SSVM causes a log exception
2012-11-16 15:38:51,752 ERROR [cloud.api.ApiDispatcher]
(Job-Executor-6:job-304) Exception while executing ExtractTemplateCmd:
com.cloud.utils.exception.CloudRuntimeException: There is no secondary
storage VM for secondary storage host 2
at
com.cloud.storage.upload.UploadMonitorImpl.createEntityDownloadURL(UploadMonitorImpl.java:209)
at
com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:426)
at
com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:294)
at
com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36)
at
com.cloud.api.commands.ExtractTemplateCmd.execute(ExtractTemplateCmd.java:126)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
But still no attempt to rebuild the SSVM or the console proxy.
--
Dave Cundiff
System Administrator
A2Hosting, Inc
http://www.a2hosting.com
Re: System VMs not rebuilding
Posted by Dave Cundiff <sy...@gmail.com>.
Hi,
Restarting the network via gui fails and generates the following error in
the gui.
Network is not in the right state to be restarted. Correct states are:
Implemented, Setup.
The following log messages are generated from the operation
2012-11-19 05:17:56,106 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-22:null) submit async job-309, details: AsyncJobVO {id:309,
userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
null, cmd: com.cloud.api.commands.RestartNetworkCmd, cmdOriginator: null,
cmdInfo:
{"id":"6b7a42dc-819f-4836-802b-b582657b0840","response":"json","cleanup":"true","sessionkey":"I8Ng6qHVKu0SQL8GJ8qkRBQM5pk\u003d","ctxUserId":"2","_":"1353320277616","ctxAccountId":"2","ctxStartEventId":"1135"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 161337998772,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2012-11-19 05:17:56,108 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-309) Executing com.cloud.api.commands.RestartNetworkCmd
for job-309
2012-11-19 05:17:56,113 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-309) Sync job-309 execution on object network.204
2012-11-19 05:17:56,248 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-309) job com.cloud.api.commands.RestartNetworkCmd for
job-309 was queued, processing the queue.
2012-11-19 05:17:56,348 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-309) Executing sync queue item: SyncQueueItemVO {id:5,
queueId: 1, contentType: AsyncJob, contentId: 309, lastProcessMsid:
161337998772, lastprocessNumber: 13, created: Mon Nov 19 05:17:56 EST 2012}
2012-11-19 05:17:56,350 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-309) Schedule queued job-309
2012-11-19 05:17:56,450 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-309) Executing
com.cloud.api.commands.RestartNetworkCmd for job-309
2012-11-19 05:17:56,451 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-9:job-309) There is a pending process in sync queue(id: 1)
2012-11-19 05:17:56,759 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-309) Complete async job-309, jobStatus: 2, resultCode:
530, result: Error Code: 431 Error text: Network is not in the right state
to be restarted. Correct states are: Implemented, Setup
2012-11-19 05:17:56,968 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-10:job-309) Sync queue (1) is currently empty
2012-11-19 05:17:56,969 WARN [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-309) Unable to unregister active job 309 from JMX
monitoring
I had rebuilt the router VMs via the destroyrouter api call earlier. It
appeared to work correctly and I currently have several virtual routers
running.
On Mon, Nov 19, 2012 at 3:32 AM, Fuchs, Andreas (SwissTXT) <
Andreas.Fuchs@swisstxt.ch> wrote:
> Dave
>
> What happens if you restart the Networks from the GUI?
>
> Regards
> Andi
>
> -----Original Message-----
> From: Dave Cundiff [mailto:syshackmin@gmail.com]
> Sent: Freitag, 16. November 2012 21:42
> To: cloudstack-users
> Subject: System VMs not rebuilding
>
> Hello,
>
> It appears the 4.0 upgrade has brought with it some bugs! I tried to
> rebuild my system VMs by erasing them via the web console. Now they just
> don't build. All I'm getting in the logs are single lines stating they
> don't exist or the zone is not ready to launch them. No attempts to build a
> VM occur. The zone/pods/hosts are all enabled.
>
> Output I'm getting in the logs.
>
>
> 2012-11-16 15:35:47,466 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.0.250/
> 2012-11-16 15:36:54,547 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
>
> Performing an operation that requires the SSVM causes a log exception
>
> 2012-11-16 15:38:51,752 ERROR [cloud.api.ApiDispatcher]
> (Job-Executor-6:job-304) Exception while executing ExtractTemplateCmd:
> com.cloud.utils.exception.CloudRuntimeException: There is no secondary
> storage VM for secondary storage host 2 at
>
> com.cloud.storage.upload.UploadMonitorImpl.createEntityDownloadURL(UploadMonitorImpl.java:209)
> at
>
> com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:426)
> at
>
> com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:294)
> at
> com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36)
> at
>
> com.cloud.api.commands.ExtractTemplateCmd.execute(ExtractTemplateCmd.java:126)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> But still no attempt to rebuild the SSVM or the console proxy.
>
>
> --
> Dave Cundiff
> System Administrator
> A2Hosting, Inc
> http://www.a2hosting.com
>
--
Dave Cundiff
System Administrator
A2Hosting, Inc
http://www.a2hosting.com
RE: System VMs not rebuilding
Posted by "Fuchs, Andreas (SwissTXT)" <An...@swisstxt.ch>.
Dave
What happens if you restart the Networks from the GUI?
Regards
Andi
-----Original Message-----
From: Dave Cundiff [mailto:syshackmin@gmail.com]
Sent: Freitag, 16. November 2012 21:42
To: cloudstack-users
Subject: System VMs not rebuilding
Hello,
It appears the 4.0 upgrade has brought with it some bugs! I tried to rebuild my system VMs by erasing them via the web console. Now they just don't build. All I'm getting in the logs are single lines stating they don't exist or the zone is not ready to launch them. No attempts to build a VM occur. The zone/pods/hosts are all enabled.
Output I'm getting in the logs.
2012-11-16 15:35:47,466 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) There is no secondary storage VM for secondary storage host nfs://10.11.0.250/
2012-11-16 15:36:54,547 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
Performing an operation that requires the SSVM causes a log exception
2012-11-16 15:38:51,752 ERROR [cloud.api.ApiDispatcher]
(Job-Executor-6:job-304) Exception while executing ExtractTemplateCmd:
com.cloud.utils.exception.CloudRuntimeException: There is no secondary storage VM for secondary storage host 2 at
com.cloud.storage.upload.UploadMonitorImpl.createEntityDownloadURL(UploadMonitorImpl.java:209)
at
com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:426)
at
com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:294)
at
com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36)
at
com.cloud.api.commands.ExtractTemplateCmd.execute(ExtractTemplateCmd.java:126)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
But still no attempt to rebuild the SSVM or the console proxy.
--
Dave Cundiff
System Administrator
A2Hosting, Inc
http://www.a2hosting.com
Re: System VMs not rebuilding
Posted by Marc Cirauqui <mc...@gmail.com>.
Is your zone enabled?
It should try to recreate SSVM and CPVM but if it's unable to do so, ther
should be something in the logs. Do you see anything about system vm not
being created?
thx
On Fri, Nov 16, 2012 at 9:41 PM, Dave Cundiff <sy...@gmail.com> wrote:
> Hello,
>
> It appears the 4.0 upgrade has brought with it some bugs! I tried to
> rebuild my system VMs by erasing them via the web console. Now they just
> don't build. All I'm getting in the logs are single lines stating they
> don't exist or the zone is not ready to launch them. No attempts to build a
> VM occur. The zone/pods/hosts are all enabled.
>
> Output I'm getting in the logs.
>
>
> 2012-11-16 15:35:47,466 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.0.250/
> 2012-11-16 15:36:54,547 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
>
> Performing an operation that requires the SSVM causes a log exception
>
> 2012-11-16 15:38:51,752 ERROR [cloud.api.ApiDispatcher]
> (Job-Executor-6:job-304) Exception while executing ExtractTemplateCmd:
> com.cloud.utils.exception.CloudRuntimeException: There is no secondary
> storage VM for secondary storage host 2
> at
>
> com.cloud.storage.upload.UploadMonitorImpl.createEntityDownloadURL(UploadMonitorImpl.java:209)
> at
>
> com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:426)
> at
>
> com.cloud.template.TemplateManagerImpl.extract(TemplateManagerImpl.java:294)
> at
> com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36)
> at
>
> com.cloud.api.commands.ExtractTemplateCmd.execute(ExtractTemplateCmd.java:126)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> But still no attempt to rebuild the SSVM or the console proxy.
>
>
> --
> Dave Cundiff
> System Administrator
> A2Hosting, Inc
> http://www.a2hosting.com
>