You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Mitch.Fiorentini" <mi...@gmail.com> on 2014/09/27 21:25:29 UTC
Re: error restarting virtual router after upgrading
Looks like I'm getting the same issue after an upgrade, this is with all my
System VMs. Normal VMs are not starting either.
2014-09-27 15:24:02,437 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-df72ecab) Exception while trying to start secondary
> storage vm
> java.lang.RuntimeException: Job failed due to exception Resource
> [Host:101] is unreachable: Host 101: Unable to start instance due to Unable
> to get answer that is of class com.cloud.agent.api.StartAnswer
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-27 15:24:02,438 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-df72ecab) Unable to start secondary storage vm for
> standby capacity, secStorageVm vm Id : 312, will recycle it and start a new
> one
On Wed, Aug 27, 2014 at 10:31 AM, Ugo Vasi <ug...@procne.it> wrote:
> Hi,
> all the “cache_mode" value were "none", in lowercase:
>
> mysql> select id,name,cache_mode from disk_offering;
> +----+------------------------------------------+------------+
> | id | name | cache_mode |
> +----+------------------------------------------+------------+
> | 1 | Small Instance | none |
> | 2 | Medium Instance | none |
> | 3 | Small | none |
> | 4 | Medium | none |
> | 5 | Large | none |
> | 6 | Custom | none |
> | 7 | System Offering For Internal LB VM | none |
> | 8 | System Offering For Software Router | none |
> | 9 | System Offering For Secondary Storage VM | none |
> | 10 | System Offering For Console Proxy | none |
> | 11 | System Offering For Elastic LB VM | none |
> +----+------------------------------------------+------------+
>
> I tried to set all uppercase:
> mysql> update disk_offering set cache_mode='NONE';
> Query OK, 11 rows affected (0.05 sec)
> Rows matched: 11 Changed: 11 Warnings: 0
>
> mysql> select id,name,cache_mode from disk_offering;
> +----+------------------------------------------+------------+
> | id | name | cache_mode |
> +----+------------------------------------------+------------+
> | 1 | Small Instance | NONE |
> | 2 | Medium Instance | NONE |
> | 3 | Small | NONE |
> | 4 | Medium | NONE |
> | 5 | Large | NONE |
> | 6 | Custom | NONE |
> | 7 | System Offering For Internal LB VM | NONE |
> | 8 | System Offering For Software Router | NONE |
> | 9 | System Offering For Secondary Storage VM | NONE |
> | 10 | System Offering For Console Proxy | NONE |
> | 11 | System Offering For Elastic LB VM | NONE |
> +----+------------------------------------------+------------+
>
> But the router is not starting. The message on web interface is:
>
> Job failed due to exception Resource [Host:4] is unreachable: Host 4:
> Unable to start instance due to Unable to get answer that is of class
> com.cloud.agent.api.StartAnswer
>
> I set the cache_mode values in the database as before, in lowercase.
>
>
>
> Il 27/08/2014 13:37, Harikrishna Patnala ha scritto:
>
> Hi,
>>
>> Can you check the value of “cache_mode" column in disk_offering table for
>> the disk_offering used by router.
>> As per the fix “cache_mode" value should be in upper case.
>>
>> -Harikrishna
>>
>> On 27-Aug-2014, at 3:00 pm, Erik Weber <te...@gmail.com> wrote:
>>
>> Most likely this: https://issues.apache.org/jira/browse/CLOUDSTACK-6893
>>>
>>> I'm not aware of a manual workaround, perhaps someone else is.
>>>
>>> --
>>> Erik
>>>
>>>
>>> On Wed, Aug 27, 2014 at 11:12 AM, Ugo Vasi <ug...@procne.it> wrote:
>>>
>>> Hi all,
>>>> we are trying to upgrade a cloustack 4.3 installation to a 4.4 version.
>>>> Manager and agents (and ssvm and cpvm) runs in a new version and seems
>>>> to
>>>> be OK but virtual routers are still 4.3.0.
>>>>
>>>> I tried to reboot a router but this does not restart it gives me these
>>>> error messages:
>>>>
>>>> ...
>>>> 2014-08-27 10:50:01,328 DEBUG [c.c.a.t.Request]
>>>> (AgentManager-Handler-5:null) Seq 1-7961519716260912076: Processing: {
>>>> Ans: , MgmtId: 240206110326297, via: 1, Ver: v1, Flags: 10,
>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum
>>>> constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.
>>>> diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer"
>>>> :{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{"
>>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{"
>>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{"
>>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{"
>>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{"
>>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com.
>>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
>>>> failure","wait":0}}] }
>>>> 2014-08-27 10:50:01,328 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-13:ctx-3a10ee96
>>>> job-120/job-121 ctx-2cd18d35) Seq 1-7961519716260912076: Received: {
>>>> Ans: ,
>>>> MgmtId: 240206110326297, via: 1, Ver: v1, Flags: 10, { Answer, Answer,
>>>> Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer,
>>>> Answer, Answer } }
>>>> 2014-08-27 10:50:01,367 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35) Failed
>>>> to start instance VM[DomainRouter|r-12-VM]
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to get answer
>>>> that is of class com.cloud.agent.api.StartAnswer
>>>> at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
>>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>> VirtualMachineManagerImpl.java:1019)
>>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>> VirtualMachineManagerImpl.java:5190)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> ...
>>>>
>>>> 2014-08-27 10:50:01,829 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35)
>>>> release
>>>> mem from host: 1, old used: 2952790016,reserved: 0, total: 50640633856;
>>>> new
>>>> used: 2818572288,reserved:0; movedfromreserved:
>>>> false,moveToReserveredfalse
>>>> 2014-08-27 10:50:01,865 ERROR [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35)
>>>> Invocation exception, caused by: com.cloud.exception.
>>>> AgentUnavailableException:
>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance due
>>>> to
>>>> Unable to get answer that is of class com.cloud.agent.api.StartAnswer
>>>> 2014-08-27 10:50:01,866 INFO [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35)
>>>> Rethrow
>>>> exception com.cloud.exception.AgentUnavailableException: Resource
>>>> [Host:1] is unreachable: Host 1: Unable to start instance due to Unable
>>>> to
>>>> get answer that is of class com.cloud.agent.api.StartAnswer
>>>> 2014-08-27 10:50:01,866 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121) Done with run of VM
>>>> work job: com.cloud.vm.VmWorkStart for VM 12, job origin: 120
>>>> 2014-08-27 10:50:01,867 ERROR [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121) Unable to complete
>>>> AsyncJobVO {id:121, userId: 2, accountId: 2, instanceType: null,
>>>> instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oA
>>>> BGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50
>>>> UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5n
>>>> L0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYX
>>>> ZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>>>> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS
>>>> 91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV2
>>>> 9ya5-
>>>> ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmR
>>>> sZXJOYW1lcQB-
>>>> AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAADHQAGVZpcnR1YWxNYWNoaW5l
>>>> TWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNo
>>>> TWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_
>>>> QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55
>>>> QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZk
>>>> bUZzZFdWNGNBRXhw,
>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>> result: null, initMsid: 240206110326297, completeMsid: null,
>>>> lastUpdated:
>>>> null, lastPolled: null, created: Wed Aug 27 10:49:59 CEST 2014}, job
>>>> origin:120
>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>> unreachable: Host 1: Unable to start instance due to Unable to get
>>>> answer
>>>> that is of class com.cloud.agent.api.StartAnswer
>>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>> VirtualMachineManagerImpl.java:1105)
>>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>> VirtualMachineManagerImpl.java:5190)
>>>> ....
>>>>
>>>> I tried to destroy the router but the new one fails to start in the same
>>>> way.
>>>>
>>>> The other router (4.3.0 version) is still working but I have not tried
>>>> to
>>>> restart it.
>>>>
>>>> Of course the vmon the network with the router ko can not get out of the
>>>> network.
>>>>
>>>> Management run on ubuntu 12.04, Agents run on ubuntu 14.04. In another
>>>> similar installation of cloudstack 4.3 it's all quite ok (I found a
>>>> problem
>>>> with a guest network, see 'Unable to start a VM due to insufficient
>>>> capacity' precedent mailing list messages).
>>>>
>>>> --
>>>>
>>>> U g o V a s i <ug...@procne.it>
>>>> P r o c n e s.r.l >)
>>>> via Cotonificio 45 33010 Tavagnacco IT
>>>> phone: +390432486523 fax: +390432486523
>>>>
>>>> Le informazioni contenute in questo messaggio sono riservate e
>>>> confidenziali ed è vietata la diffusione in qualunque modo eseguita.
>>>> Qualora Lei non fosse la persona a cui il presente messaggio è
>>>> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene
>>>> gentilmente comunicazione.
>>>> Per qualsiasi informazione si prega di contattare support@procne.it .
>>>> Rif. D.L. 196/2003
>>>>
>>>>
>>>>
>>
>>
>>
>>
>
> --
>
> U g o V a s i <ug...@procne.it>
> P r o c n e s.r.l >)
> via Cotonificio 45 33010 Tavagnacco IT
> phone: +390432486523 fax: +390432486523
>
> Le informazioni contenute in questo messaggio sono riservate e
> confidenziali ed è vietata la diffusione in qualunque modo eseguita.
> Qualora Lei non fosse la persona a cui il presente messaggio è
> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene
> gentilmente comunicazione.
> Per qualsiasi informazione si prega di contattare support@procne.it .
> Rif. D.L. 196/2003
>
>