You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/03/15 17:19:07 UTC
[URGENT - HELP NEEDED]
Hi guys,
we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
that, from CentOS 6.5 to CentOS 6.6)
And now I can not start SYSTEM VR - that is used for SHARED GUEST network
anymore.
And some VMs are down - and cant be started because they depend on this
VR...
VPC VRs are created fine, so new VR for VPC are created fine, but this one
fro Guest network fails to start:
Here you can see, after agent copies template from secondary storage, to
primary local storage, it created base image, and backing file - so storage
setup seems completed.
Than all out of sudden we have errors:
can't find connection: KVM, for vm: r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
- and BTW - absolutely NOTHING in the libvirt log !
Here is the full log, with all the steps, on the host that doesnt host any
other VMs (except one - so log is easy to read) - you can notice after 1st
fail, agent copies new image/backing file, and tries again with new image...
And BTW, same template is of course used for both System VR and VPC VRs...
2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.44.253.13/var/lib/libvirt/SECONDARY
","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
}
2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) createStoragePool didn't find existing
running pool: org.libvirt.LibvirtException: Storage pool not found: no
storage pool with matching uuid 'M2��N0�e���,�, need to create it
2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Didn't find an existing storage pool
274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
duplicate paths
2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to create storage pool
274d3297-d74e-30ce-81d7-65a3ffe52cfc
2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='netfs'>
<name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
<uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
<source>
<host name='10.44.253.13'/>
<dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
</source>
<target>
<path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
</target>
</pool>
2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <volume>
<name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing: qemu-img info
/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Execution is successful.
2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Execution is successful.
2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
}
2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
}
2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) <volume>
<name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) Execution is successful.
2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
environment variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)
Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
fmt=qcow2 size=2621440000
backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
encryption=off cluster_size=65536
2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
}
2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
}
2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Processing command:
com.cloud.agent.api.StopCommand
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: KVM, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Try to stop the vm at first
2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it
2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
}
2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) <volume>
<name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) Execution is successful.
2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
environment variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)
Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
fmt=qcow2 size=2621440000
backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
encryption=off cluster_size=65536
2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
}
2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
}
2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Processing command:
com.cloud.agent.api.StopCommand
2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find connection: KVM, for vm:
r-2281-VM, continue
2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Try to stop the vm at first
2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it
2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Nux,
no, because qemu is on exclude list in yum (I'm using intank patched
version of Centos 6.4 with RBD support) and manually compiled libvirt
1.2.3, compiled 9m ago...
Image is created, and then... just error thrown.
I also enabled debug libvirt logging and no info there eather...
Now I'm trying to downgrade packages slowly on one node...
Really crazy - VPC VR are created just fine from same template, but Guest
VR does not work...
I'm going mad slowly...
On 15 March 2015 at 19:37, Nux! <nu...@li.nux.ro> wrote:
> Hi,
>
> Has the yum update touch any of your qemu/libvirt packages?
>
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> > Sent: Sunday, 15 March, 2015 16:19:07
> > Subject: [URGENT - HELP NEEDED]
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt host
> any
> > other VMs (except one - so log is easy to read) - you can notice after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found: no
> > storage pool with matching uuid 'M2��N0�e���,�, need to create it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> > duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Nux,
no, because qemu is on exclude list in yum (I'm using intank patched
version of Centos 6.4 with RBD support) and manually compiled libvirt
1.2.3, compiled 9m ago...
Image is created, and then... just error thrown.
I also enabled debug libvirt logging and no info there eather...
Now I'm trying to downgrade packages slowly on one node...
Really crazy - VPC VR are created just fine from same template, but Guest
VR does not work...
I'm going mad slowly...
On 15 March 2015 at 19:37, Nux! <nu...@li.nux.ro> wrote:
> Hi,
>
> Has the yum update touch any of your qemu/libvirt packages?
>
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> > Sent: Sunday, 15 March, 2015 16:19:07
> > Subject: [URGENT - HELP NEEDED]
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt host
> any
> > other VMs (except one - so log is easy to read) - you can notice after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found: no
> > storage pool with matching uuid 'M2��N0�e���,�, need to create it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> > duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Hi,
Has the yum update touch any of your qemu/libvirt packages?
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> Sent: Sunday, 15 March, 2015 16:19:07
> Subject: [URGENT - HELP NEEDED]
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
>
> can't find connection: KVM, for vm: r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
>
> - and BTW - absolutely NOTHING in the libvirt log !
>
>
> Here is the full log, with all the steps, on the host that doesnt host any
> other VMs (except one - so log is easy to read) - you can notice after 1st
> fail, agent copies new image/backing file, and tries again with new image...
>
> And BTW, same template is of course used for both System VR and VPC VRs...
>
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.44.253.13/var/lib/libvirt/SECONDARY
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> }
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid 'M2��N0�e���,�, need to create it
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Didn't find an existing storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> duplicate paths
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> <source>
> <host name='10.44.253.13'/>
> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> </source>
> <target>
> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> </target>
> </pool>
>
> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <volume>
> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: qemu-img info
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) <volume>
> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Execution is successful.
> 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) <volume>
> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Try to stop the vm at first
> 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>
>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
You know I will :)
On 17 March 2015 at 00:37, Nux! <nu...@li.nux.ro> wrote:
> Andrija,
>
> Why not create a Jira issue and make some noise about it.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org
> > Cc: dev@cloudstack.apache.org
> > Sent: Monday, 16 March, 2015 21:25:24
> > Subject: Re: [URGENT - HELP NEEDED]
>
> > I would be very glad to see this patch/imrovement.
> >
> > Since it seems that I hit this problem 5-6 months ago (when we tried to
> > move instance to assign it to another account, and that failed) -but I
> > never destroyed/rebooted that VR - so maybe that was the reason I only
> > experienced problems after upgrade (VR got rebooted/recreated by the
> > script).
> >
> > Thanks Laszlo
> >
> > On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com>
> wrote:
> >
> >> Btw, just in order to avoid the same problem... maybe we could make this
> >> query smarter and if no nic marked as default, but there is only one,
> then
> >> that single NIC could be returned as default.
> >>
> >> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >>
> >> > FIXED !!!
> >> >
> >> > Thanks a lot Marcus, this is the second time you saved me from the
> deep
> >> > $$it..
> >> >
> >> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
> >> after
> >> > changing that default_nic=1, destroyed the VR, and new one was
> recreated.
> >> >
> >> > Thanks a lot for help !
> >> >
> >> >
> >> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
> >> wrote:
> >> >
> >> > > I will thanks a lot Marcus for hints...
> >> > >
> >> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >> > >
> >> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> > >> they're just triggered by you trying to fix the situation by
> >> > >> migrating, but the original issue was something else entirely.
> >> > >>
> >> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> > >
> >> > >> wrote:
> >> > >> > I did migrate and also changed accounts, unsucessfully, so some
> bugs
> >> > >> > definitively or my specific setup...
> >> > >> >
> >> > >> > Thanks, I' fixing this now and will let you know.
> >> > >> >
> >> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> > >> >
> >> > >> >> Yes, each VM should have at least one default nic, so if there's
> >> only
> >> > >> >> one nic it should be set to default. Take a db backup first, of
> >> > >> >> course, before messing with it. Any idea how it may have
> happened?
> >> Do
> >> > >> >> you migrate VMs between networks ever?
> >> > >> >>
> >> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> > >> andrija.panic@gmail.com>
> >> > >> >> wrote:
> >> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I
> need
> >> to
> >> > >> change
> >> > >> >> > all of them to default_nic=1... ?
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com>
> wrote:
> >> > >> >> >
> >> > >> >> >> VMs can have multiple nics and be on multiple networks. If
> you
> >> > set a
> >> > >> >> >> nic as default, it becomes the network that the vm has its
> >> default
> >> > >> >> >> route on. Every VM should have a default nic, and if it
> doesn't
> >> I
> >> > >> >> >> wonder how it might have happened (maybe a specific
> combination
> >> of
> >> > >> >> >> add/delete nic triggered a bug?). You should set a default
> nic
> >> for
> >> > >> >> >> every VM that might be missing one, and see if that gets your
> >> > router
> >> > >> >> >> up.
> >> > >> >> >>
> >> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> > >> andrija.panic@gmail.com
> >> > >> >> >
> >> > >> >> >> wrote:
> >> > >> >> >> > Hi Marcus,
> >> > >> >> >> >
> >> > >> >> >> > Thanks a lot fot hint
> >> > >> >> >> >
> >> > >> >> >> > True, I have the 0 as the value for some reason in
> database,
> >> for
> >> > >> >> couple
> >> > >> >> >> of
> >> > >> >> >> > NICs....
> >> > >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> > >> broadcast_uri
> >> > >> >> =
> >> > >> >> >> > "vlan://500" and default_nic = 0;
> >> > >> >> >> >
> >> > >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> > >> >> >> >
> >> > >> >> >> > or down there...
> >> > >> >> >> >
> >> > >> >> >> > This Techvee-FileServer server is already running (still
> not
> >> > dead)
> >> > >> >> and I
> >> > >> >> >> > can see 1 NIC from UI...
> >> > >> >> >> >
> >> > >> >> >> > Should I reset all of these to 1 ?
> >> > >> >> >> > What is the purpose of this field default_nic = 0.
> >> > >> >> >> >
> >> > >> >> >> > vlan://500 in my case limits results only to the network
> for
> >> > this
> >> > >> VR
> >> > >> >> that
> >> > >> >> >> > is having problems...
> >> > >> >> >> >
> >> > >> >> >> > Any suggestions ?
> >> > >> >> >> >
> >> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
> >> "netmask"
> >> > >> >> "gateway"
> >> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
> >> "strategy"
> >> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> > >> >> >> "isolation_uri"
> >> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> > >> >> "ip6_gateway"
> >> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> > >> >> "06:70:0a:00:00:ac"
> >> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-02-04
> >> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> > >> "2015-02-04
> >> > >> >> >> > 22:06:23" \N \N "0" "1"
> >> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> > >> >> "06:fc:c2:00:00:ad"
> >> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-03
> >> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> > >> "2015-03-03
> >> > >> >> >> > 14:45:47" \N \N "0" "1"
> >> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> > >> >> "06:d7:ac:00:00:ae"
> >> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-02-19
> >> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> > >> "2015-02-19
> >> > >> >> >> > 02:10:45" \N \N "0" "1"
> >> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> > >> "06:b5:72:00:00:72"
> >> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> > >> "2014-01-27
> >> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> > >> "2014-01-27
> >> > >> >> >> > 13:38:52" \N \N "0" "1"
> >> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> > >> >> "06:1d:90:00:00:72"
> >> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:15:18" \N \N "0" "1"
> >> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> > >> >> "06:33:ac:00:00:77"
> >> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> > >> "2015-03-16
> >> > >> >> >> > 11:41:11" \N \N "0" "1"
> >> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> > >> >> "06:8a:ac:00:00:77"
> >> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:02:59" \N \N "0" "1"
> >> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> > >> >> "06:c2:c4:00:00:7a"
> >> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:16:12" \N \N "0" "1"
> >> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> > >> >> "06:1c:78:00:00:7c"
> >> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:16:32" \N \N "0" "1"
> >> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> > >> "06:96:4e:00:00:8e"
> >> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-01-13
> >> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> > >> "2014-01-13
> >> > >> >> >> > 22:25:44" \N \N "0" "1"
> >> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> > >> >> "06:be:24:00:00:8e"
> >> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> >> > "2015-02-04
> >> > >> >> >> 22:54:29"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> > >> >> "06:cd:d6:00:00:97"
> >> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-09-02
> >> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> > >> "2014-09-02
> >> > >> >> >> > 09:06:56" \N \N "0" "1"
> >> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> > >> >> "06:6e:0c:00:00:9c"
> >> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> >> > "2015-02-11
> >> > >> >> >> 14:32:00"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> > >> >> "06:b8:b4:00:00:a1"
> >> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-12-02
> >> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> > >> "2014-12-02
> >> > >> >> >> > 13:58:11" \N \N "0" "1"
> >> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> > >> >> "06:f8:36:00:00:a7"
> >> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-01-21
> >> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> > >> "2015-01-21
> >> > >> >> >> > 18:46:10" \N \N "0" "1"
> >> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> > >> >> "06:ac:4e:00:00:54"
> >> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:33:59" \N \N "0" "1"
> >> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> > >> >> "06:46:d8:00:00:54"
> >> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> > >> "2014-07-23
> >> > >> >> >> > 14:54:11" \N \N "0" "1"
> >> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> > >> "06:7c:54:00:00:35"
> >> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> >> > "2015-03-16
> >> > >> >> >> 12:25:41"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> > >> >> "06:c9:e0:00:00:3a"
> >> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:03:42" \N \N "0" "1"
> >> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> > >> "06:b1:30:00:00:2a"
> >> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "206"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2013-01-11
> >> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> > >> "2013-01-11
> >> > >> >> >> > 01:11:18" \N \N "0" "1"
> >> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> > >> "06:12:88:00:00:2b"
> >> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "206"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2013-01-11
> >> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> > >> "2013-01-11
> >> > >> >> >> > 00:35:20" \N \N "0" "1"
> >> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> > >> >> "06:02:a8:00:00:e2"
> >> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> >> > "2015-02-12
> >> > >> >> >> 14:16:23"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
> >> wrote:
> >> > >> >> >> >
> >> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source
> tree
> >> > for
> >> > >> the
> >> > >> >> >> >> router code, the null pointer indicates that there's no
> >> default
> >> > >> nic
> >> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess
> that
> >> if
> >> > >> you
> >> > >> >> >> >> delete/move just that host you may be able to start the
> >> router,
> >> > >> or at
> >> > >> >> >> >> least get past this. You can also look into the db and
> see if
> >> > >> you can
> >> > >> >> >> >> find its nics in the cloud.nics table and see if there is
> in
> >> > >> fact one
> >> > >> >> >> >> marked default.
> >> > >> >> >> >>
> >> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <
> shadowsor@gmail.com
> >> >
> >> > >> wrote:
> >> > >> >> >> >> > Looks like the issue is that null pointer in
> >> CreateDhcpEntry
> >> > >> for
> >> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry
> immediately
> >> > after
> >> > >> >> that.
> >> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> > >> creating a
> >> > >> >> >> >> > DHCP entry for one of the guests serviced by this
> router.
> >> > It's
> >> > >> too
> >> > >> >> bad
> >> > >> >> >> >> > that one bad entry is fatal for the whole router.
> >> > >> >> >> >> >
> >> > >> >> >> >> >
> >> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> > >> >> >> andrija.panic@gmail.com>
> >> > >> >> >> >> wrote:
> >> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs,
> from
> >> > VPS
> >> > >> >> network
> >> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> MGMT server sends the STOP command to agent, even
> though
> >> the
> >> > >> VM
> >> > >> >> was
> >> > >> >> >> >> never
> >> > >> >> >> >> >> started, BUT the storage provisioning from template to
> >> > volume
> >> > >> is
> >> > >> >> >> done...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> We are also looking into some external help as we
> speak...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>> Hi,
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> --
> >> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> Nux!
> >> > >> >> >> >> >>> www.nux.ro
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> ----- Original Message -----
> >> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
> >> > dev@cloudstack.apache.org
> >> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> > Hi guys,
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2
> (OS
> >> > >> updated
> >> > >> >> >> right
> >> > >> >> >> >> >>> before
> >> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> >> > SHARED
> >> > >> >> GUEST
> >> > >> >> >> >> network
> >> > >> >> >> >> >>> > anymore.
> >> > >> >> >> >> >>> > And some VMs are down - and cant be started because
> >> they
> >> > >> >> depend on
> >> > >> >> >> >> this
> >> > >> >> >> >> >>> > VR...
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are
> created
> >> > >> fine,
> >> > >> >> but
> >> > >> >> >> >> this
> >> > >> >> >> >> >>> one
> >> > >> >> >> >> >>> > fro Guest network fails to start:
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > Here you can see, after agent copies template from
> >> > >> secondary
> >> > >> >> >> >> storage, to
> >> > >> >> >> >> >>> > primary local storage, it created base image, and
> >> backing
> >> > >> file
> >> > >> >> -
> >> > >> >> >> so
> >> > >> >> >> >> >>> storage
> >> > >> >> >> >> >>> > setup seems completed.
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > Than all out of sudden we have errors:
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> --
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> Andrija Panić
> >> > >> >> >> >>
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > --
> >> > >> >> >> >
> >> > >> >> >> > Andrija Panić
> >> > >> >> >>
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > --
> >> > >> >> >
> >> > >> >> > Andrija Panić
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> >
> >> > >> > Andrija Panić
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Andrija Panić
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> EOF
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
You know I will :)
On 17 March 2015 at 00:37, Nux! <nu...@li.nux.ro> wrote:
> Andrija,
>
> Why not create a Jira issue and make some noise about it.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org
> > Cc: dev@cloudstack.apache.org
> > Sent: Monday, 16 March, 2015 21:25:24
> > Subject: Re: [URGENT - HELP NEEDED]
>
> > I would be very glad to see this patch/imrovement.
> >
> > Since it seems that I hit this problem 5-6 months ago (when we tried to
> > move instance to assign it to another account, and that failed) -but I
> > never destroyed/rebooted that VR - so maybe that was the reason I only
> > experienced problems after upgrade (VR got rebooted/recreated by the
> > script).
> >
> > Thanks Laszlo
> >
> > On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com>
> wrote:
> >
> >> Btw, just in order to avoid the same problem... maybe we could make this
> >> query smarter and if no nic marked as default, but there is only one,
> then
> >> that single NIC could be returned as default.
> >>
> >> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >>
> >> > FIXED !!!
> >> >
> >> > Thanks a lot Marcus, this is the second time you saved me from the
> deep
> >> > $$it..
> >> >
> >> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
> >> after
> >> > changing that default_nic=1, destroyed the VR, and new one was
> recreated.
> >> >
> >> > Thanks a lot for help !
> >> >
> >> >
> >> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
> >> wrote:
> >> >
> >> > > I will thanks a lot Marcus for hints...
> >> > >
> >> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >> > >
> >> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> > >> they're just triggered by you trying to fix the situation by
> >> > >> migrating, but the original issue was something else entirely.
> >> > >>
> >> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> > >
> >> > >> wrote:
> >> > >> > I did migrate and also changed accounts, unsucessfully, so some
> bugs
> >> > >> > definitively or my specific setup...
> >> > >> >
> >> > >> > Thanks, I' fixing this now and will let you know.
> >> > >> >
> >> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> > >> >
> >> > >> >> Yes, each VM should have at least one default nic, so if there's
> >> only
> >> > >> >> one nic it should be set to default. Take a db backup first, of
> >> > >> >> course, before messing with it. Any idea how it may have
> happened?
> >> Do
> >> > >> >> you migrate VMs between networks ever?
> >> > >> >>
> >> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> > >> andrija.panic@gmail.com>
> >> > >> >> wrote:
> >> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I
> need
> >> to
> >> > >> change
> >> > >> >> > all of them to default_nic=1... ?
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com>
> wrote:
> >> > >> >> >
> >> > >> >> >> VMs can have multiple nics and be on multiple networks. If
> you
> >> > set a
> >> > >> >> >> nic as default, it becomes the network that the vm has its
> >> default
> >> > >> >> >> route on. Every VM should have a default nic, and if it
> doesn't
> >> I
> >> > >> >> >> wonder how it might have happened (maybe a specific
> combination
> >> of
> >> > >> >> >> add/delete nic triggered a bug?). You should set a default
> nic
> >> for
> >> > >> >> >> every VM that might be missing one, and see if that gets your
> >> > router
> >> > >> >> >> up.
> >> > >> >> >>
> >> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> > >> andrija.panic@gmail.com
> >> > >> >> >
> >> > >> >> >> wrote:
> >> > >> >> >> > Hi Marcus,
> >> > >> >> >> >
> >> > >> >> >> > Thanks a lot fot hint
> >> > >> >> >> >
> >> > >> >> >> > True, I have the 0 as the value for some reason in
> database,
> >> for
> >> > >> >> couple
> >> > >> >> >> of
> >> > >> >> >> > NICs....
> >> > >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> > >> broadcast_uri
> >> > >> >> =
> >> > >> >> >> > "vlan://500" and default_nic = 0;
> >> > >> >> >> >
> >> > >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> > >> >> >> >
> >> > >> >> >> > or down there...
> >> > >> >> >> >
> >> > >> >> >> > This Techvee-FileServer server is already running (still
> not
> >> > dead)
> >> > >> >> and I
> >> > >> >> >> > can see 1 NIC from UI...
> >> > >> >> >> >
> >> > >> >> >> > Should I reset all of these to 1 ?
> >> > >> >> >> > What is the purpose of this field default_nic = 0.
> >> > >> >> >> >
> >> > >> >> >> > vlan://500 in my case limits results only to the network
> for
> >> > this
> >> > >> VR
> >> > >> >> that
> >> > >> >> >> > is having problems...
> >> > >> >> >> >
> >> > >> >> >> > Any suggestions ?
> >> > >> >> >> >
> >> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
> >> "netmask"
> >> > >> >> "gateway"
> >> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
> >> "strategy"
> >> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> > >> >> >> "isolation_uri"
> >> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> > >> >> "ip6_gateway"
> >> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> > >> >> "06:70:0a:00:00:ac"
> >> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-02-04
> >> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> > >> "2015-02-04
> >> > >> >> >> > 22:06:23" \N \N "0" "1"
> >> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> > >> >> "06:fc:c2:00:00:ad"
> >> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-03
> >> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> > >> "2015-03-03
> >> > >> >> >> > 14:45:47" \N \N "0" "1"
> >> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> > >> >> "06:d7:ac:00:00:ae"
> >> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-02-19
> >> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> > >> "2015-02-19
> >> > >> >> >> > 02:10:45" \N \N "0" "1"
> >> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> > >> "06:b5:72:00:00:72"
> >> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> > >> "2014-01-27
> >> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> > >> "2014-01-27
> >> > >> >> >> > 13:38:52" \N \N "0" "1"
> >> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> > >> >> "06:1d:90:00:00:72"
> >> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:15:18" \N \N "0" "1"
> >> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> > >> >> "06:33:ac:00:00:77"
> >> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> > >> "2015-03-16
> >> > >> >> >> > 11:41:11" \N \N "0" "1"
> >> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> > >> >> "06:8a:ac:00:00:77"
> >> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:02:59" \N \N "0" "1"
> >> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> > >> >> "06:c2:c4:00:00:7a"
> >> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:16:12" \N \N "0" "1"
> >> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> > >> >> "06:1c:78:00:00:7c"
> >> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> > >> "2014-07-23
> >> > >> >> >> > 08:16:32" \N \N "0" "1"
> >> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> > >> "06:96:4e:00:00:8e"
> >> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-01-13
> >> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> > >> "2014-01-13
> >> > >> >> >> > 22:25:44" \N \N "0" "1"
> >> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> > >> >> "06:be:24:00:00:8e"
> >> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> >> > "2015-02-04
> >> > >> >> >> 22:54:29"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> > >> >> "06:cd:d6:00:00:97"
> >> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-09-02
> >> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> > >> "2014-09-02
> >> > >> >> >> > 09:06:56" \N \N "0" "1"
> >> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> > >> >> "06:6e:0c:00:00:9c"
> >> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> >> > "2015-02-11
> >> > >> >> >> 14:32:00"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> > >> >> "06:b8:b4:00:00:a1"
> >> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-12-02
> >> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> > >> "2014-12-02
> >> > >> >> >> > 13:58:11" \N \N "0" "1"
> >> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> > >> >> "06:f8:36:00:00:a7"
> >> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-01-21
> >> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> > >> "2015-01-21
> >> > >> >> >> > 18:46:10" \N \N "0" "1"
> >> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> > >> >> "06:ac:4e:00:00:54"
> >> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> > >> "2014-07-23
> >> > >> >> >> > 10:33:59" \N \N "0" "1"
> >> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> > >> >> "06:46:d8:00:00:54"
> >> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2014-07-23
> >> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> > >> "2014-07-23
> >> > >> >> >> > 14:54:11" \N \N "0" "1"
> >> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> > >> "06:7c:54:00:00:35"
> >> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> >> > "2015-03-16
> >> > >> >> >> 12:25:41"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> > >> >> "06:c9:e0:00:00:3a"
> >> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2015-03-16
> >> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> > >> "2015-03-16
> >> > >> >> >> > 12:03:42" \N \N "0" "1"
> >> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> > >> "06:b1:30:00:00:2a"
> >> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "206"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2013-01-11
> >> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> > >> "2013-01-11
> >> > >> >> >> > 01:11:18" \N \N "0" "1"
> >> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> > >> "06:12:88:00:00:2b"
> >> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "206"
> >> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> > >> "2013-01-11
> >> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> > >> "2013-01-11
> >> > >> >> >> > 00:35:20" \N \N "0" "1"
> >> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> > >> >> "06:02:a8:00:00:e2"
> >> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> >> > "vlan://500"
> >> > >> >> "212"
> >> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> >> > "2015-02-12
> >> > >> >> >> 14:16:23"
> >> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N
> "0"
> >> > "1"
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
> >> wrote:
> >> > >> >> >> >
> >> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source
> tree
> >> > for
> >> > >> the
> >> > >> >> >> >> router code, the null pointer indicates that there's no
> >> default
> >> > >> nic
> >> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess
> that
> >> if
> >> > >> you
> >> > >> >> >> >> delete/move just that host you may be able to start the
> >> router,
> >> > >> or at
> >> > >> >> >> >> least get past this. You can also look into the db and
> see if
> >> > >> you can
> >> > >> >> >> >> find its nics in the cloud.nics table and see if there is
> in
> >> > >> fact one
> >> > >> >> >> >> marked default.
> >> > >> >> >> >>
> >> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <
> shadowsor@gmail.com
> >> >
> >> > >> wrote:
> >> > >> >> >> >> > Looks like the issue is that null pointer in
> >> CreateDhcpEntry
> >> > >> for
> >> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry
> immediately
> >> > after
> >> > >> >> that.
> >> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> > >> creating a
> >> > >> >> >> >> > DHCP entry for one of the guests serviced by this
> router.
> >> > It's
> >> > >> too
> >> > >> >> bad
> >> > >> >> >> >> > that one bad entry is fatal for the whole router.
> >> > >> >> >> >> >
> >> > >> >> >> >> >
> >> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> > >> >> >> andrija.panic@gmail.com>
> >> > >> >> >> >> wrote:
> >> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs,
> from
> >> > VPS
> >> > >> >> network
> >> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> MGMT server sends the STOP command to agent, even
> though
> >> the
> >> > >> VM
> >> > >> >> was
> >> > >> >> >> >> never
> >> > >> >> >> >> >> started, BUT the storage provisioning from template to
> >> > volume
> >> > >> is
> >> > >> >> >> done...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> We are also looking into some external help as we
> speak...
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>> Hi,
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> --
> >> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> Nux!
> >> > >> >> >> >> >>> www.nux.ro
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> ----- Original Message -----
> >> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
> >> > dev@cloudstack.apache.org
> >> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>> > Hi guys,
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2
> (OS
> >> > >> updated
> >> > >> >> >> right
> >> > >> >> >> >> >>> before
> >> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> >> > SHARED
> >> > >> >> GUEST
> >> > >> >> >> >> network
> >> > >> >> >> >> >>> > anymore.
> >> > >> >> >> >> >>> > And some VMs are down - and cant be started because
> >> they
> >> > >> >> depend on
> >> > >> >> >> >> this
> >> > >> >> >> >> >>> > VR...
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are
> created
> >> > >> fine,
> >> > >> >> but
> >> > >> >> >> >> this
> >> > >> >> >> >> >>> one
> >> > >> >> >> >> >>> > fro Guest network fails to start:
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > Here you can see, after agent copies template from
> >> > >> secondary
> >> > >> >> >> >> storage, to
> >> > >> >> >> >> >>> > primary local storage, it created base image, and
> >> backing
> >> > >> file
> >> > >> >> -
> >> > >> >> >> so
> >> > >> >> >> >> >>> storage
> >> > >> >> >> >> >>> > setup seems completed.
> >> > >> >> >> >> >>> >
> >> > >> >> >> >> >>> > Than all out of sudden we have errors:
> >> > >> >> >> >> >>>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> --
> >> > >> >> >> >> >>
> >> > >> >> >> >> >> Andrija Panić
> >> > >> >> >> >>
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > --
> >> > >> >> >> >
> >> > >> >> >> > Andrija Panić
> >> > >> >> >>
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > --
> >> > >> >> >
> >> > >> >> > Andrija Panić
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> >
> >> > >> > Andrija Panić
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Andrija Panić
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> EOF
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Andrija,
Why not create a Jira issue and make some noise about it.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Sent: Monday, 16 March, 2015 21:25:24
> Subject: Re: [URGENT - HELP NEEDED]
> I would be very glad to see this patch/imrovement.
>
> Since it seems that I hit this problem 5-6 months ago (when we tried to
> move instance to assign it to another account, and that failed) -but I
> never destroyed/rebooted that VR - so maybe that was the reason I only
> experienced problems after upgrade (VR got rebooted/recreated by the
> script).
>
> Thanks Laszlo
>
> On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com> wrote:
>
>> Btw, just in order to avoid the same problem... maybe we could make this
>> query smarter and if no nic marked as default, but there is only one, then
>> that single NIC could be returned as default.
>>
>> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
>> wrote:
>>
>> > FIXED !!!
>> >
>> > Thanks a lot Marcus, this is the second time you saved me from the deep
>> > $$it..
>> >
>> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
>> after
>> > changing that default_nic=1, destroyed the VR, and new one was recreated.
>> >
>> > Thanks a lot for help !
>> >
>> >
>> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
>> wrote:
>> >
>> > > I will thanks a lot Marcus for hints...
>> > >
>> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>> > >
>> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>> > >> they're just triggered by you trying to fix the situation by
>> > >> migrating, but the original issue was something else entirely.
>> > >>
>> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
>> andrija.panic@gmail.com
>> > >
>> > >> wrote:
>> > >> > I did migrate and also changed accounts, unsucessfully, so some bugs
>> > >> > definitively or my specific setup...
>> > >> >
>> > >> > Thanks, I' fixing this now and will let you know.
>> > >> >
>> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>> > >> >
>> > >> >> Yes, each VM should have at least one default nic, so if there's
>> only
>> > >> >> one nic it should be set to default. Take a db backup first, of
>> > >> >> course, before messing with it. Any idea how it may have happened?
>> Do
>> > >> >> you migrate VMs between networks ever?
>> > >> >>
>> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>> > >> andrija.panic@gmail.com>
>> > >> >> wrote:
>> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need
>> to
>> > >> change
>> > >> >> > all of them to default_nic=1... ?
>> > >> >> >
>> > >> >> >
>> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> > >> >> >
>> > >> >> >> VMs can have multiple nics and be on multiple networks. If you
>> > set a
>> > >> >> >> nic as default, it becomes the network that the vm has its
>> default
>> > >> >> >> route on. Every VM should have a default nic, and if it doesn't
>> I
>> > >> >> >> wonder how it might have happened (maybe a specific combination
>> of
>> > >> >> >> add/delete nic triggered a bug?). You should set a default nic
>> for
>> > >> >> >> every VM that might be missing one, and see if that gets your
>> > router
>> > >> >> >> up.
>> > >> >> >>
>> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>> > >> andrija.panic@gmail.com
>> > >> >> >
>> > >> >> >> wrote:
>> > >> >> >> > Hi Marcus,
>> > >> >> >> >
>> > >> >> >> > Thanks a lot fot hint
>> > >> >> >> >
>> > >> >> >> > True, I have the 0 as the value for some reason in database,
>> for
>> > >> >> couple
>> > >> >> >> of
>> > >> >> >> > NICs....
>> > >> >> >> > select * from nics where ip4_address like "46.232%" and
>> > >> broadcast_uri
>> > >> >> =
>> > >> >> >> > "vlan://500" and default_nic = 0;
>> > >> >> >> >
>> > >> >> >> > results: http://pastebin.com/rDAe2RY9
>> > >> >> >> >
>> > >> >> >> > or down there...
>> > >> >> >> >
>> > >> >> >> > This Techvee-FileServer server is already running (still not
>> > dead)
>> > >> >> and I
>> > >> >> >> > can see 1 NIC from UI...
>> > >> >> >> >
>> > >> >> >> > Should I reset all of these to 1 ?
>> > >> >> >> > What is the purpose of this field default_nic = 0.
>> > >> >> >> >
>> > >> >> >> > vlan://500 in my case limits results only to the network for
>> > this
>> > >> VR
>> > >> >> that
>> > >> >> >> > is having problems...
>> > >> >> >> >
>> > >> >> >> > Any suggestions ?
>> > >> >> >> >
>> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
>> "netmask"
>> > >> >> "gateway"
>> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
>> "strategy"
>> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> > >> >> >> "isolation_uri"
>> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> > >> >> "ip6_gateway"
>> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> > >> >> "06:70:0a:00:00:ac"
>> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-02-04
>> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>> > >> "2015-02-04
>> > >> >> >> > 22:06:23" \N \N "0" "1"
>> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> > >> >> "06:fc:c2:00:00:ad"
>> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-03
>> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>> > >> "2015-03-03
>> > >> >> >> > 14:45:47" \N \N "0" "1"
>> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> > >> >> "06:d7:ac:00:00:ae"
>> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-02-19
>> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>> > >> "2015-02-19
>> > >> >> >> > 02:10:45" \N \N "0" "1"
>> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>> > >> "06:b5:72:00:00:72"
>> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>> > >> "2014-01-27
>> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>> > >> "2014-01-27
>> > >> >> >> > 13:38:52" \N \N "0" "1"
>> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> > >> >> "06:1d:90:00:00:72"
>> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>> > >> "2014-07-23
>> > >> >> >> > 08:15:18" \N \N "0" "1"
>> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> > >> >> "06:33:ac:00:00:77"
>> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>> > >> "2015-03-16
>> > >> >> >> > 11:41:11" \N \N "0" "1"
>> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> > >> >> "06:8a:ac:00:00:77"
>> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>> > >> "2015-03-16
>> > >> >> >> > 12:02:59" \N \N "0" "1"
>> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> > >> >> "06:c2:c4:00:00:7a"
>> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>> > >> "2014-07-23
>> > >> >> >> > 08:16:12" \N \N "0" "1"
>> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> > >> >> "06:1c:78:00:00:7c"
>> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>> > >> "2014-07-23
>> > >> >> >> > 08:16:32" \N \N "0" "1"
>> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>> > >> "06:96:4e:00:00:8e"
>> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-01-13
>> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>> > >> "2014-01-13
>> > >> >> >> > 22:25:44" \N \N "0" "1"
>> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> > >> >> "06:be:24:00:00:8e"
>> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
>> > "2015-02-04
>> > >> >> >> 22:54:29"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
>> > "1"
>> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> > >> >> "06:cd:d6:00:00:97"
>> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-09-02
>> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>> > >> "2014-09-02
>> > >> >> >> > 09:06:56" \N \N "0" "1"
>> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> > >> >> "06:6e:0c:00:00:9c"
>> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
>> > "2015-02-11
>> > >> >> >> 14:32:00"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
>> > "1"
>> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> > >> >> "06:b8:b4:00:00:a1"
>> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-12-02
>> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>> > >> "2014-12-02
>> > >> >> >> > 13:58:11" \N \N "0" "1"
>> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> > >> >> "06:f8:36:00:00:a7"
>> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-01-21
>> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>> > >> "2015-01-21
>> > >> >> >> > 18:46:10" \N \N "0" "1"
>> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> > >> >> "06:ac:4e:00:00:54"
>> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>> > >> "2014-07-23
>> > >> >> >> > 10:33:59" \N \N "0" "1"
>> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> > >> >> "06:46:d8:00:00:54"
>> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>> > >> "2014-07-23
>> > >> >> >> > 14:54:11" \N \N "0" "1"
>> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>> > >> "06:7c:54:00:00:35"
>> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
>> > "2015-03-16
>> > >> >> >> 12:25:41"
>> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
>> > "1"
>> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> > >> >> "06:c9:e0:00:00:3a"
>> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>> > >> "2015-03-16
>> > >> >> >> > 12:03:42" \N \N "0" "1"
>> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>> > >> "06:b1:30:00:00:2a"
>> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "206"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2013-01-11
>> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>> > >> "2013-01-11
>> > >> >> >> > 01:11:18" \N \N "0" "1"
>> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>> > >> "06:12:88:00:00:2b"
>> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "206"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2013-01-11
>> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>> > >> "2013-01-11
>> > >> >> >> > 00:35:20" \N \N "0" "1"
>> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> > >> >> "06:02:a8:00:00:e2"
>> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
>> > "2015-02-12
>> > >> >> >> 14:16:23"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
>> > "1"
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
>> wrote:
>> > >> >> >> >
>> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
>> > for
>> > >> the
>> > >> >> >> >> router code, the null pointer indicates that there's no
>> default
>> > >> nic
>> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that
>> if
>> > >> you
>> > >> >> >> >> delete/move just that host you may be able to start the
>> router,
>> > >> or at
>> > >> >> >> >> least get past this. You can also look into the db and see if
>> > >> you can
>> > >> >> >> >> find its nics in the cloud.nics table and see if there is in
>> > >> fact one
>> > >> >> >> >> marked default.
>> > >> >> >> >>
>> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <shadowsor@gmail.com
>> >
>> > >> wrote:
>> > >> >> >> >> > Looks like the issue is that null pointer in
>> CreateDhcpEntry
>> > >> for
>> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
>> > after
>> > >> >> that.
>> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
>> > >> creating a
>> > >> >> >> >> > DHCP entry for one of the guests serviced by this router.
>> > It's
>> > >> too
>> > >> >> bad
>> > >> >> >> >> > that one bad entry is fatal for the whole router.
>> > >> >> >> >> >
>> > >> >> >> >> >
>> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> > >> >> >> andrija.panic@gmail.com>
>> > >> >> >> >> wrote:
>> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
>> > VPS
>> > >> >> network
>> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
>> > >> >> >> >> >>
>> > >> >> >> >> >> MGMT server sends the STOP command to agent, even though
>> the
>> > >> VM
>> > >> >> was
>> > >> >> >> >> never
>> > >> >> >> >> >> started, BUT the storage provisioning from template to
>> > volume
>> > >> is
>> > >> >> >> done...
>> > >> >> >> >> >>
>> > >> >> >> >> >> We are also looking into some external help as we speak...
>> > >> >> >> >> >>
>> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> > >> >> >> >> >>
>> > >> >> >> >> >>> Hi,
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> Have you managed to get to the bottom of this?
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> --
>> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> Nux!
>> > >> >> >> >> >>> www.nux.ro
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> ----- Original Message -----
>> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
>> > dev@cloudstack.apache.org
>> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> > Hi guys,
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>> > >> updated
>> > >> >> >> right
>> > >> >> >> >> >>> before
>> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
>> > SHARED
>> > >> >> GUEST
>> > >> >> >> >> network
>> > >> >> >> >> >>> > anymore.
>> > >> >> >> >> >>> > And some VMs are down - and cant be started because
>> they
>> > >> >> depend on
>> > >> >> >> >> this
>> > >> >> >> >> >>> > VR...
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>> > >> fine,
>> > >> >> but
>> > >> >> >> >> this
>> > >> >> >> >> >>> one
>> > >> >> >> >> >>> > fro Guest network fails to start:
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > Here you can see, after agent copies template from
>> > >> secondary
>> > >> >> >> >> storage, to
>> > >> >> >> >> >>> > primary local storage, it created base image, and
>> backing
>> > >> file
>> > >> >> -
>> > >> >> >> so
>> > >> >> >> >> >>> storage
>> > >> >> >> >> >>> > setup seems completed.
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > Than all out of sudden we have errors:
>> > >> >> >> >> >>>
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >> --
>> > >> >> >> >> >>
>> > >> >> >> >> >> Andrija Panić
>> > >> >> >> >>
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > --
>> > >> >> >> >
>> > >> >> >> > Andrija Panić
>> > >> >> >>
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> > --
>> > >> >> >
>> > >> >> > Andrija Panić
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> >
>> > >> > Andrija Panić
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Andrija Panić
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>> >
>>
>>
>>
>> --
>>
>> EOF
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Andrija,
Why not create a Jira issue and make some noise about it.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Sent: Monday, 16 March, 2015 21:25:24
> Subject: Re: [URGENT - HELP NEEDED]
> I would be very glad to see this patch/imrovement.
>
> Since it seems that I hit this problem 5-6 months ago (when we tried to
> move instance to assign it to another account, and that failed) -but I
> never destroyed/rebooted that VR - so maybe that was the reason I only
> experienced problems after upgrade (VR got rebooted/recreated by the
> script).
>
> Thanks Laszlo
>
> On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com> wrote:
>
>> Btw, just in order to avoid the same problem... maybe we could make this
>> query smarter and if no nic marked as default, but there is only one, then
>> that single NIC could be returned as default.
>>
>> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
>> wrote:
>>
>> > FIXED !!!
>> >
>> > Thanks a lot Marcus, this is the second time you saved me from the deep
>> > $$it..
>> >
>> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
>> after
>> > changing that default_nic=1, destroyed the VR, and new one was recreated.
>> >
>> > Thanks a lot for help !
>> >
>> >
>> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
>> wrote:
>> >
>> > > I will thanks a lot Marcus for hints...
>> > >
>> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>> > >
>> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>> > >> they're just triggered by you trying to fix the situation by
>> > >> migrating, but the original issue was something else entirely.
>> > >>
>> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
>> andrija.panic@gmail.com
>> > >
>> > >> wrote:
>> > >> > I did migrate and also changed accounts, unsucessfully, so some bugs
>> > >> > definitively or my specific setup...
>> > >> >
>> > >> > Thanks, I' fixing this now and will let you know.
>> > >> >
>> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>> > >> >
>> > >> >> Yes, each VM should have at least one default nic, so if there's
>> only
>> > >> >> one nic it should be set to default. Take a db backup first, of
>> > >> >> course, before messing with it. Any idea how it may have happened?
>> Do
>> > >> >> you migrate VMs between networks ever?
>> > >> >>
>> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>> > >> andrija.panic@gmail.com>
>> > >> >> wrote:
>> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need
>> to
>> > >> change
>> > >> >> > all of them to default_nic=1... ?
>> > >> >> >
>> > >> >> >
>> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> > >> >> >
>> > >> >> >> VMs can have multiple nics and be on multiple networks. If you
>> > set a
>> > >> >> >> nic as default, it becomes the network that the vm has its
>> default
>> > >> >> >> route on. Every VM should have a default nic, and if it doesn't
>> I
>> > >> >> >> wonder how it might have happened (maybe a specific combination
>> of
>> > >> >> >> add/delete nic triggered a bug?). You should set a default nic
>> for
>> > >> >> >> every VM that might be missing one, and see if that gets your
>> > router
>> > >> >> >> up.
>> > >> >> >>
>> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>> > >> andrija.panic@gmail.com
>> > >> >> >
>> > >> >> >> wrote:
>> > >> >> >> > Hi Marcus,
>> > >> >> >> >
>> > >> >> >> > Thanks a lot fot hint
>> > >> >> >> >
>> > >> >> >> > True, I have the 0 as the value for some reason in database,
>> for
>> > >> >> couple
>> > >> >> >> of
>> > >> >> >> > NICs....
>> > >> >> >> > select * from nics where ip4_address like "46.232%" and
>> > >> broadcast_uri
>> > >> >> =
>> > >> >> >> > "vlan://500" and default_nic = 0;
>> > >> >> >> >
>> > >> >> >> > results: http://pastebin.com/rDAe2RY9
>> > >> >> >> >
>> > >> >> >> > or down there...
>> > >> >> >> >
>> > >> >> >> > This Techvee-FileServer server is already running (still not
>> > dead)
>> > >> >> and I
>> > >> >> >> > can see 1 NIC from UI...
>> > >> >> >> >
>> > >> >> >> > Should I reset all of these to 1 ?
>> > >> >> >> > What is the purpose of this field default_nic = 0.
>> > >> >> >> >
>> > >> >> >> > vlan://500 in my case limits results only to the network for
>> > this
>> > >> VR
>> > >> >> that
>> > >> >> >> > is having problems...
>> > >> >> >> >
>> > >> >> >> > Any suggestions ?
>> > >> >> >> >
>> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
>> "netmask"
>> > >> >> "gateway"
>> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
>> "strategy"
>> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> > >> >> >> "isolation_uri"
>> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> > >> >> "ip6_gateway"
>> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> > >> >> "06:70:0a:00:00:ac"
>> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-02-04
>> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>> > >> "2015-02-04
>> > >> >> >> > 22:06:23" \N \N "0" "1"
>> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> > >> >> "06:fc:c2:00:00:ad"
>> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-03
>> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>> > >> "2015-03-03
>> > >> >> >> > 14:45:47" \N \N "0" "1"
>> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> > >> >> "06:d7:ac:00:00:ae"
>> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-02-19
>> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>> > >> "2015-02-19
>> > >> >> >> > 02:10:45" \N \N "0" "1"
>> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>> > >> "06:b5:72:00:00:72"
>> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>> > >> "2014-01-27
>> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>> > >> "2014-01-27
>> > >> >> >> > 13:38:52" \N \N "0" "1"
>> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> > >> >> "06:1d:90:00:00:72"
>> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>> > >> "2014-07-23
>> > >> >> >> > 08:15:18" \N \N "0" "1"
>> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> > >> >> "06:33:ac:00:00:77"
>> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>> > >> "2015-03-16
>> > >> >> >> > 11:41:11" \N \N "0" "1"
>> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> > >> >> "06:8a:ac:00:00:77"
>> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>> > >> "2015-03-16
>> > >> >> >> > 12:02:59" \N \N "0" "1"
>> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> > >> >> "06:c2:c4:00:00:7a"
>> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>> > >> "2014-07-23
>> > >> >> >> > 08:16:12" \N \N "0" "1"
>> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> > >> >> "06:1c:78:00:00:7c"
>> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>> > >> "2014-07-23
>> > >> >> >> > 08:16:32" \N \N "0" "1"
>> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>> > >> "06:96:4e:00:00:8e"
>> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-01-13
>> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>> > >> "2014-01-13
>> > >> >> >> > 22:25:44" \N \N "0" "1"
>> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> > >> >> "06:be:24:00:00:8e"
>> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
>> > "2015-02-04
>> > >> >> >> 22:54:29"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
>> > "1"
>> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> > >> >> "06:cd:d6:00:00:97"
>> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-09-02
>> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>> > >> "2014-09-02
>> > >> >> >> > 09:06:56" \N \N "0" "1"
>> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> > >> >> "06:6e:0c:00:00:9c"
>> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
>> > "2015-02-11
>> > >> >> >> 14:32:00"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
>> > "1"
>> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> > >> >> "06:b8:b4:00:00:a1"
>> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-12-02
>> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>> > >> "2014-12-02
>> > >> >> >> > 13:58:11" \N \N "0" "1"
>> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> > >> >> "06:f8:36:00:00:a7"
>> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-01-21
>> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>> > >> "2015-01-21
>> > >> >> >> > 18:46:10" \N \N "0" "1"
>> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> > >> >> "06:ac:4e:00:00:54"
>> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>> > >> "2014-07-23
>> > >> >> >> > 10:33:59" \N \N "0" "1"
>> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> > >> >> "06:46:d8:00:00:54"
>> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2014-07-23
>> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>> > >> "2014-07-23
>> > >> >> >> > 14:54:11" \N \N "0" "1"
>> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>> > >> "06:7c:54:00:00:35"
>> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
>> > "2015-03-16
>> > >> >> >> 12:25:41"
>> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
>> > "1"
>> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> > >> >> "06:c9:e0:00:00:3a"
>> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2015-03-16
>> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>> > >> "2015-03-16
>> > >> >> >> > 12:03:42" \N \N "0" "1"
>> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>> > >> "06:b1:30:00:00:2a"
>> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "206"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2013-01-11
>> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>> > >> "2013-01-11
>> > >> >> >> > 01:11:18" \N \N "0" "1"
>> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>> > >> "06:12:88:00:00:2b"
>> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "206"
>> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> > >> "2013-01-11
>> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>> > >> "2013-01-11
>> > >> >> >> > 00:35:20" \N \N "0" "1"
>> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> > >> >> "06:02:a8:00:00:e2"
>> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
>> > "vlan://500"
>> > >> >> "212"
>> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
>> > "2015-02-12
>> > >> >> >> 14:16:23"
>> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
>> > "1"
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
>> wrote:
>> > >> >> >> >
>> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
>> > for
>> > >> the
>> > >> >> >> >> router code, the null pointer indicates that there's no
>> default
>> > >> nic
>> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that
>> if
>> > >> you
>> > >> >> >> >> delete/move just that host you may be able to start the
>> router,
>> > >> or at
>> > >> >> >> >> least get past this. You can also look into the db and see if
>> > >> you can
>> > >> >> >> >> find its nics in the cloud.nics table and see if there is in
>> > >> fact one
>> > >> >> >> >> marked default.
>> > >> >> >> >>
>> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <shadowsor@gmail.com
>> >
>> > >> wrote:
>> > >> >> >> >> > Looks like the issue is that null pointer in
>> CreateDhcpEntry
>> > >> for
>> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
>> > after
>> > >> >> that.
>> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
>> > >> creating a
>> > >> >> >> >> > DHCP entry for one of the guests serviced by this router.
>> > It's
>> > >> too
>> > >> >> bad
>> > >> >> >> >> > that one bad entry is fatal for the whole router.
>> > >> >> >> >> >
>> > >> >> >> >> >
>> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> > >> >> >> andrija.panic@gmail.com>
>> > >> >> >> >> wrote:
>> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
>> > VPS
>> > >> >> network
>> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
>> > >> >> >> >> >>
>> > >> >> >> >> >> MGMT server sends the STOP command to agent, even though
>> the
>> > >> VM
>> > >> >> was
>> > >> >> >> >> never
>> > >> >> >> >> >> started, BUT the storage provisioning from template to
>> > volume
>> > >> is
>> > >> >> >> done...
>> > >> >> >> >> >>
>> > >> >> >> >> >> We are also looking into some external help as we speak...
>> > >> >> >> >> >>
>> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> > >> >> >> >> >>
>> > >> >> >> >> >>> Hi,
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> Have you managed to get to the bottom of this?
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> --
>> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> Nux!
>> > >> >> >> >> >>> www.nux.ro
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> ----- Original Message -----
>> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
>> > dev@cloudstack.apache.org
>> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> > >> >> >> >> >>>
>> > >> >> >> >> >>> > Hi guys,
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>> > >> updated
>> > >> >> >> right
>> > >> >> >> >> >>> before
>> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
>> > SHARED
>> > >> >> GUEST
>> > >> >> >> >> network
>> > >> >> >> >> >>> > anymore.
>> > >> >> >> >> >>> > And some VMs are down - and cant be started because
>> they
>> > >> >> depend on
>> > >> >> >> >> this
>> > >> >> >> >> >>> > VR...
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>> > >> fine,
>> > >> >> but
>> > >> >> >> >> this
>> > >> >> >> >> >>> one
>> > >> >> >> >> >>> > fro Guest network fails to start:
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > Here you can see, after agent copies template from
>> > >> secondary
>> > >> >> >> >> storage, to
>> > >> >> >> >> >>> > primary local storage, it created base image, and
>> backing
>> > >> file
>> > >> >> -
>> > >> >> >> so
>> > >> >> >> >> >>> storage
>> > >> >> >> >> >>> > setup seems completed.
>> > >> >> >> >> >>> >
>> > >> >> >> >> >>> > Than all out of sudden we have errors:
>> > >> >> >> >> >>>
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >> --
>> > >> >> >> >> >>
>> > >> >> >> >> >> Andrija Panić
>> > >> >> >> >>
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > --
>> > >> >> >> >
>> > >> >> >> > Andrija Panić
>> > >> >> >>
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> > --
>> > >> >> >
>> > >> >> > Andrija Panić
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> >
>> > >> > Andrija Panić
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Andrija Panić
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>> >
>>
>>
>>
>> --
>>
>> EOF
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Cool, congrats! :-)
On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
> > I will thanks a lot Marcus for hints...
> >
> > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >
> >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> they're just triggered by you trying to fix the situation by
> >> migrating, but the original issue was something else entirely.
> >>
> >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> >> > definitively or my specific setup...
> >> >
> >> > Thanks, I' fixing this now and will let you know.
> >> >
> >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> Yes, each VM should have at least one default nic, so if there's only
> >> >> one nic it should be set to default. Take a db backup first, of
> >> >> course, before messing with it. Any idea how it may have happened? Do
> >> >> you migrate VMs between networks ever?
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> >> change
> >> >> > all of them to default_nic=1... ?
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> VMs can have multiple nics and be on multiple networks. If you
> set a
> >> >> >> nic as default, it becomes the network that the vm has its default
> >> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> >> every VM that might be missing one, and see if that gets your
> router
> >> >> >> up.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hi Marcus,
> >> >> >> >
> >> >> >> > Thanks a lot fot hint
> >> >> >> >
> >> >> >> > True, I have the 0 as the value for some reason in database, for
> >> >> couple
> >> >> >> of
> >> >> >> > NICs....
> >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> broadcast_uri
> >> >> =
> >> >> >> > "vlan://500" and default_nic = 0;
> >> >> >> >
> >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >> >
> >> >> >> > or down there...
> >> >> >> >
> >> >> >> > This Techvee-FileServer server is already running (still not
> dead)
> >> >> and I
> >> >> >> > can see 1 NIC from UI...
> >> >> >> >
> >> >> >> > Should I reset all of these to 1 ?
> >> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >> >
> >> >> >> > vlan://500 in my case limits results only to the network for
> this
> >> VR
> >> >> that
> >> >> >> > is having problems...
> >> >> >> >
> >> >> >> > Any suggestions ?
> >> >> >> >
> >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> >> "gateway"
> >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> >> "isolation_uri"
> >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> >> "ip6_gateway"
> >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> >> "06:70:0a:00:00:ac"
> >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-04
> >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> "2015-02-04
> >> >> >> > 22:06:23" \N \N "0" "1"
> >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> >> "06:fc:c2:00:00:ad"
> >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-03
> >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> "2015-03-03
> >> >> >> > 14:45:47" \N \N "0" "1"
> >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> >> "06:d7:ac:00:00:ae"
> >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-19
> >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> "2015-02-19
> >> >> >> > 02:10:45" \N \N "0" "1"
> >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> "06:b5:72:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> "2014-01-27
> >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> "2014-01-27
> >> >> >> > 13:38:52" \N \N "0" "1"
> >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> >> "06:1d:90:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> "2014-07-23
> >> >> >> > 08:15:18" \N \N "0" "1"
> >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> >> "06:33:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> "2015-03-16
> >> >> >> > 11:41:11" \N \N "0" "1"
> >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> >> "06:8a:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> "2015-03-16
> >> >> >> > 12:02:59" \N \N "0" "1"
> >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> >> "06:c2:c4:00:00:7a"
> >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> "2014-07-23
> >> >> >> > 08:16:12" \N \N "0" "1"
> >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> >> "06:1c:78:00:00:7c"
> >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> "2014-07-23
> >> >> >> > 08:16:32" \N \N "0" "1"
> >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> "06:96:4e:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-01-13
> >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> "2014-01-13
> >> >> >> > 22:25:44" \N \N "0" "1"
> >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> >> "06:be:24:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> >> 22:54:29"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> "1"
> >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> >> "06:cd:d6:00:00:97"
> >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-09-02
> >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> "2014-09-02
> >> >> >> > 09:06:56" \N \N "0" "1"
> >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> >> "06:6e:0c:00:00:9c"
> >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> "2015-02-11
> >> >> >> 14:32:00"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> "1"
> >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> >> "06:b8:b4:00:00:a1"
> >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-12-02
> >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> "2014-12-02
> >> >> >> > 13:58:11" \N \N "0" "1"
> >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> >> "06:f8:36:00:00:a7"
> >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-01-21
> >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> "2015-01-21
> >> >> >> > 18:46:10" \N \N "0" "1"
> >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> >> "06:ac:4e:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> "2014-07-23
> >> >> >> > 10:33:59" \N \N "0" "1"
> >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> >> "06:46:d8:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> "2014-07-23
> >> >> >> > 14:54:11" \N \N "0" "1"
> >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> "06:7c:54:00:00:35"
> >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> "2015-03-16
> >> >> >> 12:25:41"
> >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> "1"
> >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> >> "06:c9:e0:00:00:3a"
> >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> "2015-03-16
> >> >> >> > 12:03:42" \N \N "0" "1"
> >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> "06:b1:30:00:00:2a"
> >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> "2013-01-11
> >> >> >> > 01:11:18" \N \N "0" "1"
> >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> "06:12:88:00:00:2b"
> >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> "2013-01-11
> >> >> >> > 00:35:20" \N \N "0" "1"
> >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> >> "06:02:a8:00:00:e2"
> >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-12
> >> >> >> 14:16:23"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> "1"
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> for
> >> the
> >> >> >> >> router code, the null pointer indicates that there's no default
> >> nic
> >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
> >> you
> >> >> >> >> delete/move just that host you may be able to start the router,
> >> or at
> >> >> >> >> least get past this. You can also look into the db and see if
> >> you can
> >> >> >> >> find its nics in the cloud.nics table and see if there is in
> >> fact one
> >> >> >> >> marked default.
> >> >> >> >>
> >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> >> wrote:
> >> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
> >> for
> >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> after
> >> >> that.
> >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> creating a
> >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> It's
> >> too
> >> >> bad
> >> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> >> andrija.panic@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> VPS
> >> >> network
> >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >> >>
> >> >> >> >> >> MGMT server sends the STOP command to agent, even though the
> >> VM
> >> >> was
> >> >> >> >> never
> >> >> >> >> >> started, BUT the storage provisioning from template to
> volume
> >> is
> >> >> >> done...
> >> >> >> >> >>
> >> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >> >>
> >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi,
> >> >> >> >> >>>
> >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >> >>>
> >> >> >> >> >>> Nux!
> >> >> >> >> >>> www.nux.ro
> >> >> >> >> >>>
> >> >> >> >> >>> ----- Original Message -----
> >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >> >>> > To: users@cloudstack.apache.org,
> dev@cloudstack.apache.org
> >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >> >>>
> >> >> >> >> >>> > Hi guys,
> >> >> >> >> >>> >
> >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> >> updated
> >> >> >> right
> >> >> >> >> >>> before
> >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >> >>> >
> >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> SHARED
> >> >> GUEST
> >> >> >> >> network
> >> >> >> >> >>> > anymore.
> >> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> >> depend on
> >> >> >> >> this
> >> >> >> >> >>> > VR...
> >> >> >> >> >>> >
> >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> >> fine,
> >> >> but
> >> >> >> >> this
> >> >> >> >> >>> one
> >> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >> >>> >
> >> >> >> >> >>> > Here you can see, after agent copies template from
> >> secondary
> >> >> >> >> storage, to
> >> >> >> >> >>> > primary local storage, it created base image, and backing
> >> file
> >> >> -
> >> >> >> so
> >> >> >> >> >>> storage
> >> >> >> >> >>> > setup seems completed.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >>
> >> >> >> >> >> Andrija Panić
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Good to hear. Glad I could lend another pair of eyes.
On Mon, Mar 16, 2015 at 9:19 AM, Andrija Panic <an...@gmail.com> wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
>> I will thanks a lot Marcus for hints...
>>
>> On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>>
>>> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>>> they're just triggered by you trying to fix the situation by
>>> migrating, but the original issue was something else entirely.
>>>
>>> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
>>> wrote:
>>> > I did migrate and also changed accounts, unsucessfully, so some bugs
>>> > definitively or my specific setup...
>>> >
>>> > Thanks, I' fixing this now and will let you know.
>>> >
>>> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>>> >
>>> >> Yes, each VM should have at least one default nic, so if there's only
>>> >> one nic it should be set to default. Take a db backup first, of
>>> >> course, before messing with it. Any idea how it may have happened? Do
>>> >> you migrate VMs between networks ever?
>>> >>
>>> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>>> andrija.panic@gmail.com>
>>> >> wrote:
>>> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
>>> change
>>> >> > all of them to default_nic=1... ?
>>> >> >
>>> >> >
>>> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>>> >> >
>>> >> >> VMs can have multiple nics and be on multiple networks. If you set a
>>> >> >> nic as default, it becomes the network that the vm has its default
>>> >> >> route on. Every VM should have a default nic, and if it doesn't I
>>> >> >> wonder how it might have happened (maybe a specific combination of
>>> >> >> add/delete nic triggered a bug?). You should set a default nic for
>>> >> >> every VM that might be missing one, and see if that gets your router
>>> >> >> up.
>>> >> >>
>>> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>>> andrija.panic@gmail.com
>>> >> >
>>> >> >> wrote:
>>> >> >> > Hi Marcus,
>>> >> >> >
>>> >> >> > Thanks a lot fot hint
>>> >> >> >
>>> >> >> > True, I have the 0 as the value for some reason in database, for
>>> >> couple
>>> >> >> of
>>> >> >> > NICs....
>>> >> >> > select * from nics where ip4_address like "46.232%" and
>>> broadcast_uri
>>> >> =
>>> >> >> > "vlan://500" and default_nic = 0;
>>> >> >> >
>>> >> >> > results: http://pastebin.com/rDAe2RY9
>>> >> >> >
>>> >> >> > or down there...
>>> >> >> >
>>> >> >> > This Techvee-FileServer server is already running (still not dead)
>>> >> and I
>>> >> >> > can see 1 NIC from UI...
>>> >> >> >
>>> >> >> > Should I reset all of these to 1 ?
>>> >> >> > What is the purpose of this field default_nic = 0.
>>> >> >> >
>>> >> >> > vlan://500 in my case limits results only to the network for this
>>> VR
>>> >> that
>>> >> >> > is having problems...
>>> >> >> >
>>> >> >> > Any suggestions ?
>>> >> >> >
>>> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>>> >> "gateway"
>>> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>>> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>>> >> >> "isolation_uri"
>>> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>>> >> "ip6_gateway"
>>> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>>> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>>> >> "06:70:0a:00:00:ac"
>>> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-02-04
>>> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>>> "2015-02-04
>>> >> >> > 22:06:23" \N \N "0" "1"
>>> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>>> >> "06:fc:c2:00:00:ad"
>>> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-03
>>> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>>> "2015-03-03
>>> >> >> > 14:45:47" \N \N "0" "1"
>>> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>>> >> "06:d7:ac:00:00:ae"
>>> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-02-19
>>> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>>> "2015-02-19
>>> >> >> > 02:10:45" \N \N "0" "1"
>>> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>>> "06:b5:72:00:00:72"
>>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>>> "2014-01-27
>>> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>>> "2014-01-27
>>> >> >> > 13:38:52" \N \N "0" "1"
>>> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>>> >> "06:1d:90:00:00:72"
>>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>>> "2014-07-23
>>> >> >> > 08:15:18" \N \N "0" "1"
>>> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>>> >> "06:33:ac:00:00:77"
>>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>>> "2015-03-16
>>> >> >> > 11:41:11" \N \N "0" "1"
>>> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>>> >> "06:8a:ac:00:00:77"
>>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>>> "2015-03-16
>>> >> >> > 12:02:59" \N \N "0" "1"
>>> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>>> >> "06:c2:c4:00:00:7a"
>>> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>>> "2014-07-23
>>> >> >> > 08:16:12" \N \N "0" "1"
>>> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>>> >> "06:1c:78:00:00:7c"
>>> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>>> "2014-07-23
>>> >> >> > 08:16:32" \N \N "0" "1"
>>> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>>> "06:96:4e:00:00:8e"
>>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-01-13
>>> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>>> "2014-01-13
>>> >> >> > 22:25:44" \N \N "0" "1"
>>> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>>> >> "06:be:24:00:00:8e"
>>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>>> >> >> 22:54:29"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>>> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>>> >> "06:cd:d6:00:00:97"
>>> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-09-02
>>> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>>> "2014-09-02
>>> >> >> > 09:06:56" \N \N "0" "1"
>>> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>>> >> "06:6e:0c:00:00:9c"
>>> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>>> >> >> 14:32:00"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>>> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>>> >> "06:b8:b4:00:00:a1"
>>> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-12-02
>>> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>>> "2014-12-02
>>> >> >> > 13:58:11" \N \N "0" "1"
>>> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>>> >> "06:f8:36:00:00:a7"
>>> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-01-21
>>> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>>> "2015-01-21
>>> >> >> > 18:46:10" \N \N "0" "1"
>>> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>>> >> "06:ac:4e:00:00:54"
>>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>>> "2014-07-23
>>> >> >> > 10:33:59" \N \N "0" "1"
>>> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>>> >> "06:46:d8:00:00:54"
>>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>>> "2014-07-23
>>> >> >> > 14:54:11" \N \N "0" "1"
>>> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>>> "06:7c:54:00:00:35"
>>> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>>> >> >> 12:25:41"
>>> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>>> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>>> >> "06:c9:e0:00:00:3a"
>>> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>>> "2015-03-16
>>> >> >> > 12:03:42" \N \N "0" "1"
>>> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>>> "06:b1:30:00:00:2a"
>>> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "206"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2013-01-11
>>> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>>> "2013-01-11
>>> >> >> > 01:11:18" \N \N "0" "1"
>>> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>>> "06:12:88:00:00:2b"
>>> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "206"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2013-01-11
>>> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>>> "2013-01-11
>>> >> >> > 00:35:20" \N \N "0" "1"
>>> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>>> >> "06:02:a8:00:00:e2"
>>> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>>> >> >> 14:16:23"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>>> >> >> >
>>> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
>>> the
>>> >> >> >> router code, the null pointer indicates that there's no default
>>> nic
>>> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
>>> you
>>> >> >> >> delete/move just that host you may be able to start the router,
>>> or at
>>> >> >> >> least get past this. You can also look into the db and see if
>>> you can
>>> >> >> >> find its nics in the cloud.nics table and see if there is in
>>> fact one
>>> >> >> >> marked default.
>>> >> >> >>
>>> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
>>> wrote:
>>> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
>>> for
>>> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>>> >> that.
>>> >> >> >> > It would suggest some inconsistent/unexpected data when
>>> creating a
>>> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
>>> too
>>> >> bad
>>> >> >> >> > that one bad entry is fatal for the whole router.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>>> >> >> andrija.panic@gmail.com>
>>> >> >> >> wrote:
>>> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>>> >> network
>>> >> >> >> >> (Guest Shared netwotk) to VPCs...
>>> >> >> >> >>
>>> >> >> >> >> MGMT server sends the STOP command to agent, even though the
>>> VM
>>> >> was
>>> >> >> >> never
>>> >> >> >> >> started, BUT the storage provisioning from template to volume
>>> is
>>> >> >> done...
>>> >> >> >> >>
>>> >> >> >> >> We are also looking into some external help as we speak...
>>> >> >> >> >>
>>> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>>> >> >> >> >>
>>> >> >> >> >>> Hi,
>>> >> >> >> >>>
>>> >> >> >> >>> Have you managed to get to the bottom of this?
>>> >> >> >> >>>
>>> >> >> >> >>> --
>>> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>>> >> >> >> >>>
>>> >> >> >> >>> Nux!
>>> >> >> >> >>> www.nux.ro
>>> >> >> >> >>>
>>> >> >> >> >>> ----- Original Message -----
>>> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>>> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>>> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>>> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>>> >> >> >> >>>
>>> >> >> >> >>> > Hi guys,
>>> >> >> >> >>> >
>>> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>>> updated
>>> >> >> right
>>> >> >> >> >>> before
>>> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>>> >> >> >> >>> >
>>> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>>> >> GUEST
>>> >> >> >> network
>>> >> >> >> >>> > anymore.
>>> >> >> >> >>> > And some VMs are down - and cant be started because they
>>> >> depend on
>>> >> >> >> this
>>> >> >> >> >>> > VR...
>>> >> >> >> >>> >
>>> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>>> fine,
>>> >> but
>>> >> >> >> this
>>> >> >> >> >>> one
>>> >> >> >> >>> > fro Guest network fails to start:
>>> >> >> >> >>> >
>>> >> >> >> >>> > Here you can see, after agent copies template from
>>> secondary
>>> >> >> >> storage, to
>>> >> >> >> >>> > primary local storage, it created base image, and backing
>>> file
>>> >> -
>>> >> >> so
>>> >> >> >> >>> storage
>>> >> >> >> >>> > setup seems completed.
>>> >> >> >> >>> >
>>> >> >> >> >>> > Than all out of sudden we have errors:
>>> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> --
>>> >> >> >> >>
>>> >> >> >> >> Andrija Panić
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> >
>>> >> >> > Andrija Panić
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Andrija Panić
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Cool, congrats! :-)
On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
> > I will thanks a lot Marcus for hints...
> >
> > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >
> >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> they're just triggered by you trying to fix the situation by
> >> migrating, but the original issue was something else entirely.
> >>
> >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> >> > definitively or my specific setup...
> >> >
> >> > Thanks, I' fixing this now and will let you know.
> >> >
> >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> Yes, each VM should have at least one default nic, so if there's only
> >> >> one nic it should be set to default. Take a db backup first, of
> >> >> course, before messing with it. Any idea how it may have happened? Do
> >> >> you migrate VMs between networks ever?
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> >> change
> >> >> > all of them to default_nic=1... ?
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> VMs can have multiple nics and be on multiple networks. If you
> set a
> >> >> >> nic as default, it becomes the network that the vm has its default
> >> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> >> every VM that might be missing one, and see if that gets your
> router
> >> >> >> up.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hi Marcus,
> >> >> >> >
> >> >> >> > Thanks a lot fot hint
> >> >> >> >
> >> >> >> > True, I have the 0 as the value for some reason in database, for
> >> >> couple
> >> >> >> of
> >> >> >> > NICs....
> >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> broadcast_uri
> >> >> =
> >> >> >> > "vlan://500" and default_nic = 0;
> >> >> >> >
> >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >> >
> >> >> >> > or down there...
> >> >> >> >
> >> >> >> > This Techvee-FileServer server is already running (still not
> dead)
> >> >> and I
> >> >> >> > can see 1 NIC from UI...
> >> >> >> >
> >> >> >> > Should I reset all of these to 1 ?
> >> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >> >
> >> >> >> > vlan://500 in my case limits results only to the network for
> this
> >> VR
> >> >> that
> >> >> >> > is having problems...
> >> >> >> >
> >> >> >> > Any suggestions ?
> >> >> >> >
> >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> >> "gateway"
> >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> >> "isolation_uri"
> >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> >> "ip6_gateway"
> >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> >> "06:70:0a:00:00:ac"
> >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-04
> >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> "2015-02-04
> >> >> >> > 22:06:23" \N \N "0" "1"
> >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> >> "06:fc:c2:00:00:ad"
> >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-03
> >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> "2015-03-03
> >> >> >> > 14:45:47" \N \N "0" "1"
> >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> >> "06:d7:ac:00:00:ae"
> >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-19
> >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> "2015-02-19
> >> >> >> > 02:10:45" \N \N "0" "1"
> >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> "06:b5:72:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> "2014-01-27
> >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> "2014-01-27
> >> >> >> > 13:38:52" \N \N "0" "1"
> >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> >> "06:1d:90:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> "2014-07-23
> >> >> >> > 08:15:18" \N \N "0" "1"
> >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> >> "06:33:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> "2015-03-16
> >> >> >> > 11:41:11" \N \N "0" "1"
> >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> >> "06:8a:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> "2015-03-16
> >> >> >> > 12:02:59" \N \N "0" "1"
> >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> >> "06:c2:c4:00:00:7a"
> >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> "2014-07-23
> >> >> >> > 08:16:12" \N \N "0" "1"
> >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> >> "06:1c:78:00:00:7c"
> >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> "2014-07-23
> >> >> >> > 08:16:32" \N \N "0" "1"
> >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> "06:96:4e:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-01-13
> >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> "2014-01-13
> >> >> >> > 22:25:44" \N \N "0" "1"
> >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> >> "06:be:24:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> >> 22:54:29"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> "1"
> >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> >> "06:cd:d6:00:00:97"
> >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-09-02
> >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> "2014-09-02
> >> >> >> > 09:06:56" \N \N "0" "1"
> >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> >> "06:6e:0c:00:00:9c"
> >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> "2015-02-11
> >> >> >> 14:32:00"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> "1"
> >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> >> "06:b8:b4:00:00:a1"
> >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-12-02
> >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> "2014-12-02
> >> >> >> > 13:58:11" \N \N "0" "1"
> >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> >> "06:f8:36:00:00:a7"
> >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-01-21
> >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> "2015-01-21
> >> >> >> > 18:46:10" \N \N "0" "1"
> >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> >> "06:ac:4e:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> "2014-07-23
> >> >> >> > 10:33:59" \N \N "0" "1"
> >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> >> "06:46:d8:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> "2014-07-23
> >> >> >> > 14:54:11" \N \N "0" "1"
> >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> "06:7c:54:00:00:35"
> >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> "2015-03-16
> >> >> >> 12:25:41"
> >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> "1"
> >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> >> "06:c9:e0:00:00:3a"
> >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> "2015-03-16
> >> >> >> > 12:03:42" \N \N "0" "1"
> >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> "06:b1:30:00:00:2a"
> >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> "2013-01-11
> >> >> >> > 01:11:18" \N \N "0" "1"
> >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> "06:12:88:00:00:2b"
> >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> "2013-01-11
> >> >> >> > 00:35:20" \N \N "0" "1"
> >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> >> "06:02:a8:00:00:e2"
> >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-12
> >> >> >> 14:16:23"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> "1"
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> for
> >> the
> >> >> >> >> router code, the null pointer indicates that there's no default
> >> nic
> >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
> >> you
> >> >> >> >> delete/move just that host you may be able to start the router,
> >> or at
> >> >> >> >> least get past this. You can also look into the db and see if
> >> you can
> >> >> >> >> find its nics in the cloud.nics table and see if there is in
> >> fact one
> >> >> >> >> marked default.
> >> >> >> >>
> >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> >> wrote:
> >> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
> >> for
> >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> after
> >> >> that.
> >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> creating a
> >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> It's
> >> too
> >> >> bad
> >> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> >> andrija.panic@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> VPS
> >> >> network
> >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >> >>
> >> >> >> >> >> MGMT server sends the STOP command to agent, even though the
> >> VM
> >> >> was
> >> >> >> >> never
> >> >> >> >> >> started, BUT the storage provisioning from template to
> volume
> >> is
> >> >> >> done...
> >> >> >> >> >>
> >> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >> >>
> >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi,
> >> >> >> >> >>>
> >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >> >>>
> >> >> >> >> >>> Nux!
> >> >> >> >> >>> www.nux.ro
> >> >> >> >> >>>
> >> >> >> >> >>> ----- Original Message -----
> >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >> >>> > To: users@cloudstack.apache.org,
> dev@cloudstack.apache.org
> >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >> >>>
> >> >> >> >> >>> > Hi guys,
> >> >> >> >> >>> >
> >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> >> updated
> >> >> >> right
> >> >> >> >> >>> before
> >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >> >>> >
> >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> SHARED
> >> >> GUEST
> >> >> >> >> network
> >> >> >> >> >>> > anymore.
> >> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> >> depend on
> >> >> >> >> this
> >> >> >> >> >>> > VR...
> >> >> >> >> >>> >
> >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> >> fine,
> >> >> but
> >> >> >> >> this
> >> >> >> >> >>> one
> >> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >> >>> >
> >> >> >> >> >>> > Here you can see, after agent copies template from
> >> secondary
> >> >> >> >> storage, to
> >> >> >> >> >>> > primary local storage, it created base image, and backing
> >> file
> >> >> -
> >> >> >> so
> >> >> >> >> >>> storage
> >> >> >> >> >>> > setup seems completed.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >>
> >> >> >> >> >> Andrija Panić
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I would be very glad to see this patch/imrovement.
Since it seems that I hit this problem 5-6 months ago (when we tried to
move instance to assign it to another account, and that failed) -but I
never destroyed/rebooted that VR - so maybe that was the reason I only
experienced problems after upgrade (VR got rebooted/recreated by the
script).
Thanks Laszlo
On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com> wrote:
> Btw, just in order to avoid the same problem... maybe we could make this
> query smarter and if no nic marked as default, but there is only one, then
> that single NIC could be returned as default.
>
> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > FIXED !!!
> >
> > Thanks a lot Marcus, this is the second time you saved me from the deep
> > $$it..
> >
> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
> after
> > changing that default_nic=1, destroyed the VR, and new one was recreated.
> >
> > Thanks a lot for help !
> >
> >
> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
> wrote:
> >
> > > I will thanks a lot Marcus for hints...
> > >
> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> > >
> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> > >> they're just triggered by you trying to fix the situation by
> > >> migrating, but the original issue was something else entirely.
> > >>
> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
> andrija.panic@gmail.com
> > >
> > >> wrote:
> > >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> > >> > definitively or my specific setup...
> > >> >
> > >> > Thanks, I' fixing this now and will let you know.
> > >> >
> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> > >> >
> > >> >> Yes, each VM should have at least one default nic, so if there's
> only
> > >> >> one nic it should be set to default. Take a db backup first, of
> > >> >> course, before messing with it. Any idea how it may have happened?
> Do
> > >> >> you migrate VMs between networks ever?
> > >> >>
> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> > >> andrija.panic@gmail.com>
> > >> >> wrote:
> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need
> to
> > >> change
> > >> >> > all of them to default_nic=1... ?
> > >> >> >
> > >> >> >
> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> > >> >> >
> > >> >> >> VMs can have multiple nics and be on multiple networks. If you
> > set a
> > >> >> >> nic as default, it becomes the network that the vm has its
> default
> > >> >> >> route on. Every VM should have a default nic, and if it doesn't
> I
> > >> >> >> wonder how it might have happened (maybe a specific combination
> of
> > >> >> >> add/delete nic triggered a bug?). You should set a default nic
> for
> > >> >> >> every VM that might be missing one, and see if that gets your
> > router
> > >> >> >> up.
> > >> >> >>
> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> > >> andrija.panic@gmail.com
> > >> >> >
> > >> >> >> wrote:
> > >> >> >> > Hi Marcus,
> > >> >> >> >
> > >> >> >> > Thanks a lot fot hint
> > >> >> >> >
> > >> >> >> > True, I have the 0 as the value for some reason in database,
> for
> > >> >> couple
> > >> >> >> of
> > >> >> >> > NICs....
> > >> >> >> > select * from nics where ip4_address like "46.232%" and
> > >> broadcast_uri
> > >> >> =
> > >> >> >> > "vlan://500" and default_nic = 0;
> > >> >> >> >
> > >> >> >> > results: http://pastebin.com/rDAe2RY9
> > >> >> >> >
> > >> >> >> > or down there...
> > >> >> >> >
> > >> >> >> > This Techvee-FileServer server is already running (still not
> > dead)
> > >> >> and I
> > >> >> >> > can see 1 NIC from UI...
> > >> >> >> >
> > >> >> >> > Should I reset all of these to 1 ?
> > >> >> >> > What is the purpose of this field default_nic = 0.
> > >> >> >> >
> > >> >> >> > vlan://500 in my case limits results only to the network for
> > this
> > >> VR
> > >> >> that
> > >> >> >> > is having problems...
> > >> >> >> >
> > >> >> >> > Any suggestions ?
> > >> >> >> >
> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
> "netmask"
> > >> >> "gateway"
> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
> "strategy"
> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> > >> >> >> "isolation_uri"
> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> > >> >> "ip6_gateway"
> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> > >> >> "06:70:0a:00:00:ac"
> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-02-04
> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> > >> "2015-02-04
> > >> >> >> > 22:06:23" \N \N "0" "1"
> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> > >> >> "06:fc:c2:00:00:ad"
> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-03
> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> > >> "2015-03-03
> > >> >> >> > 14:45:47" \N \N "0" "1"
> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> > >> >> "06:d7:ac:00:00:ae"
> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-02-19
> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> > >> "2015-02-19
> > >> >> >> > 02:10:45" \N \N "0" "1"
> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> > >> "06:b5:72:00:00:72"
> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> > >> "2014-01-27
> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> > >> "2014-01-27
> > >> >> >> > 13:38:52" \N \N "0" "1"
> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> > >> >> "06:1d:90:00:00:72"
> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> > >> "2014-07-23
> > >> >> >> > 08:15:18" \N \N "0" "1"
> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> > >> >> "06:33:ac:00:00:77"
> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> > >> "2015-03-16
> > >> >> >> > 11:41:11" \N \N "0" "1"
> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> > >> >> "06:8a:ac:00:00:77"
> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> > >> "2015-03-16
> > >> >> >> > 12:02:59" \N \N "0" "1"
> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> > >> >> "06:c2:c4:00:00:7a"
> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> > >> "2014-07-23
> > >> >> >> > 08:16:12" \N \N "0" "1"
> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> > >> >> "06:1c:78:00:00:7c"
> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> > >> "2014-07-23
> > >> >> >> > 08:16:32" \N \N "0" "1"
> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> > >> "06:96:4e:00:00:8e"
> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-01-13
> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> > >> "2014-01-13
> > >> >> >> > 22:25:44" \N \N "0" "1"
> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> > >> >> "06:be:24:00:00:8e"
> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> > "2015-02-04
> > >> >> >> 22:54:29"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> > "1"
> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> > >> >> "06:cd:d6:00:00:97"
> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-09-02
> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> > >> "2014-09-02
> > >> >> >> > 09:06:56" \N \N "0" "1"
> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> > >> >> "06:6e:0c:00:00:9c"
> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> > "2015-02-11
> > >> >> >> 14:32:00"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> > "1"
> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> > >> >> "06:b8:b4:00:00:a1"
> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-12-02
> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> > >> "2014-12-02
> > >> >> >> > 13:58:11" \N \N "0" "1"
> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> > >> >> "06:f8:36:00:00:a7"
> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-01-21
> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> > >> "2015-01-21
> > >> >> >> > 18:46:10" \N \N "0" "1"
> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> > >> >> "06:ac:4e:00:00:54"
> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> > >> "2014-07-23
> > >> >> >> > 10:33:59" \N \N "0" "1"
> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> > >> >> "06:46:d8:00:00:54"
> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> > >> "2014-07-23
> > >> >> >> > 14:54:11" \N \N "0" "1"
> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> > >> "06:7c:54:00:00:35"
> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> > "2015-03-16
> > >> >> >> 12:25:41"
> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> > "1"
> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> > >> >> "06:c9:e0:00:00:3a"
> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> > >> "2015-03-16
> > >> >> >> > 12:03:42" \N \N "0" "1"
> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> > >> "06:b1:30:00:00:2a"
> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "206"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2013-01-11
> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> > >> "2013-01-11
> > >> >> >> > 01:11:18" \N \N "0" "1"
> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> > >> "06:12:88:00:00:2b"
> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "206"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2013-01-11
> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> > >> "2013-01-11
> > >> >> >> > 00:35:20" \N \N "0" "1"
> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> > >> >> "06:02:a8:00:00:e2"
> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> > "2015-02-12
> > >> >> >> 14:16:23"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> > "1"
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
> wrote:
> > >> >> >> >
> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> > for
> > >> the
> > >> >> >> >> router code, the null pointer indicates that there's no
> default
> > >> nic
> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that
> if
> > >> you
> > >> >> >> >> delete/move just that host you may be able to start the
> router,
> > >> or at
> > >> >> >> >> least get past this. You can also look into the db and see if
> > >> you can
> > >> >> >> >> find its nics in the cloud.nics table and see if there is in
> > >> fact one
> > >> >> >> >> marked default.
> > >> >> >> >>
> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <shadowsor@gmail.com
> >
> > >> wrote:
> > >> >> >> >> > Looks like the issue is that null pointer in
> CreateDhcpEntry
> > >> for
> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> > after
> > >> >> that.
> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
> > >> creating a
> > >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> > It's
> > >> too
> > >> >> bad
> > >> >> >> >> > that one bad entry is fatal for the whole router.
> > >> >> >> >> >
> > >> >> >> >> >
> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> > >> >> >> andrija.panic@gmail.com>
> > >> >> >> >> wrote:
> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> > VPS
> > >> >> network
> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> > >> >> >> >> >>
> > >> >> >> >> >> MGMT server sends the STOP command to agent, even though
> the
> > >> VM
> > >> >> was
> > >> >> >> >> never
> > >> >> >> >> >> started, BUT the storage provisioning from template to
> > volume
> > >> is
> > >> >> >> done...
> > >> >> >> >> >>
> > >> >> >> >> >> We are also looking into some external help as we speak...
> > >> >> >> >> >>
> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> > >> >> >> >> >>
> > >> >> >> >> >>> Hi,
> > >> >> >> >> >>>
> > >> >> >> >> >>> Have you managed to get to the bottom of this?
> > >> >> >> >> >>>
> > >> >> >> >> >>> --
> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> > >> >> >> >> >>>
> > >> >> >> >> >>> Nux!
> > >> >> >> >> >>> www.nux.ro
> > >> >> >> >> >>>
> > >> >> >> >> >>> ----- Original Message -----
> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
> > dev@cloudstack.apache.org
> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> > >> >> >> >> >>>
> > >> >> >> >> >>> > Hi guys,
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> > >> updated
> > >> >> >> right
> > >> >> >> >> >>> before
> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> > SHARED
> > >> >> GUEST
> > >> >> >> >> network
> > >> >> >> >> >>> > anymore.
> > >> >> >> >> >>> > And some VMs are down - and cant be started because
> they
> > >> >> depend on
> > >> >> >> >> this
> > >> >> >> >> >>> > VR...
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> > >> fine,
> > >> >> but
> > >> >> >> >> this
> > >> >> >> >> >>> one
> > >> >> >> >> >>> > fro Guest network fails to start:
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > Here you can see, after agent copies template from
> > >> secondary
> > >> >> >> >> storage, to
> > >> >> >> >> >>> > primary local storage, it created base image, and
> backing
> > >> file
> > >> >> -
> > >> >> >> so
> > >> >> >> >> >>> storage
> > >> >> >> >> >>> > setup seems completed.
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > Than all out of sudden we have errors:
> > >> >> >> >> >>>
> > >> >> >> >> >>
> > >> >> >> >> >>
> > >> >> >> >> >>
> > >> >> >> >> >> --
> > >> >> >> >> >>
> > >> >> >> >> >> Andrija Panić
> > >> >> >> >>
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > --
> > >> >> >> >
> > >> >> >> > Andrija Panić
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> >
> > >> >> > Andrija Panić
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrija Panić
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> EOF
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I would be very glad to see this patch/imrovement.
Since it seems that I hit this problem 5-6 months ago (when we tried to
move instance to assign it to another account, and that failed) -but I
never destroyed/rebooted that VR - so maybe that was the reason I only
experienced problems after upgrade (VR got rebooted/recreated by the
script).
Thanks Laszlo
On 16 March 2015 at 18:42, Laszlo Hornyak <la...@gmail.com> wrote:
> Btw, just in order to avoid the same problem... maybe we could make this
> query smarter and if no nic marked as default, but there is only one, then
> that single NIC could be returned as default.
>
> On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > FIXED !!!
> >
> > Thanks a lot Marcus, this is the second time you saved me from the deep
> > $$it..
> >
> > Only 1 VM that had only 1 NIC and not set to default in DB - so just
> after
> > changing that default_nic=1, destroyed the VR, and new one was recreated.
> >
> > Thanks a lot for help !
> >
> >
> > On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com>
> wrote:
> >
> > > I will thanks a lot Marcus for hints...
> > >
> > > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> > >
> > >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> > >> they're just triggered by you trying to fix the situation by
> > >> migrating, but the original issue was something else entirely.
> > >>
> > >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <
> andrija.panic@gmail.com
> > >
> > >> wrote:
> > >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> > >> > definitively or my specific setup...
> > >> >
> > >> > Thanks, I' fixing this now and will let you know.
> > >> >
> > >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> > >> >
> > >> >> Yes, each VM should have at least one default nic, so if there's
> only
> > >> >> one nic it should be set to default. Take a db backup first, of
> > >> >> course, before messing with it. Any idea how it may have happened?
> Do
> > >> >> you migrate VMs between networks ever?
> > >> >>
> > >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> > >> andrija.panic@gmail.com>
> > >> >> wrote:
> > >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need
> to
> > >> change
> > >> >> > all of them to default_nic=1... ?
> > >> >> >
> > >> >> >
> > >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> > >> >> >
> > >> >> >> VMs can have multiple nics and be on multiple networks. If you
> > set a
> > >> >> >> nic as default, it becomes the network that the vm has its
> default
> > >> >> >> route on. Every VM should have a default nic, and if it doesn't
> I
> > >> >> >> wonder how it might have happened (maybe a specific combination
> of
> > >> >> >> add/delete nic triggered a bug?). You should set a default nic
> for
> > >> >> >> every VM that might be missing one, and see if that gets your
> > router
> > >> >> >> up.
> > >> >> >>
> > >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> > >> andrija.panic@gmail.com
> > >> >> >
> > >> >> >> wrote:
> > >> >> >> > Hi Marcus,
> > >> >> >> >
> > >> >> >> > Thanks a lot fot hint
> > >> >> >> >
> > >> >> >> > True, I have the 0 as the value for some reason in database,
> for
> > >> >> couple
> > >> >> >> of
> > >> >> >> > NICs....
> > >> >> >> > select * from nics where ip4_address like "46.232%" and
> > >> broadcast_uri
> > >> >> =
> > >> >> >> > "vlan://500" and default_nic = 0;
> > >> >> >> >
> > >> >> >> > results: http://pastebin.com/rDAe2RY9
> > >> >> >> >
> > >> >> >> > or down there...
> > >> >> >> >
> > >> >> >> > This Techvee-FileServer server is already running (still not
> > dead)
> > >> >> and I
> > >> >> >> > can see 1 NIC from UI...
> > >> >> >> >
> > >> >> >> > Should I reset all of these to 1 ?
> > >> >> >> > What is the purpose of this field default_nic = 0.
> > >> >> >> >
> > >> >> >> > vlan://500 in my case limits results only to the network for
> > this
> > >> VR
> > >> >> that
> > >> >> >> > is having problems...
> > >> >> >> >
> > >> >> >> > Any suggestions ?
> > >> >> >> >
> > >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address"
> "netmask"
> > >> >> "gateway"
> > >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state"
> "strategy"
> > >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> > >> >> >> "isolation_uri"
> > >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> > >> >> "ip6_gateway"
> > >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> > >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> > >> >> "06:70:0a:00:00:ac"
> > >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-02-04
> > >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> > >> "2015-02-04
> > >> >> >> > 22:06:23" \N \N "0" "1"
> > >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> > >> >> "06:fc:c2:00:00:ad"
> > >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-03
> > >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> > >> "2015-03-03
> > >> >> >> > 14:45:47" \N \N "0" "1"
> > >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> > >> >> "06:d7:ac:00:00:ae"
> > >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-02-19
> > >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> > >> "2015-02-19
> > >> >> >> > 02:10:45" \N \N "0" "1"
> > >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> > >> "06:b5:72:00:00:72"
> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> > >> "2014-01-27
> > >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> > >> "2014-01-27
> > >> >> >> > 13:38:52" \N \N "0" "1"
> > >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> > >> >> "06:1d:90:00:00:72"
> > >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> > >> "2014-07-23
> > >> >> >> > 08:15:18" \N \N "0" "1"
> > >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> > >> >> "06:33:ac:00:00:77"
> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> > >> "2015-03-16
> > >> >> >> > 11:41:11" \N \N "0" "1"
> > >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> > >> >> "06:8a:ac:00:00:77"
> > >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> > >> "2015-03-16
> > >> >> >> > 12:02:59" \N \N "0" "1"
> > >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> > >> >> "06:c2:c4:00:00:7a"
> > >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> > >> "2014-07-23
> > >> >> >> > 08:16:12" \N \N "0" "1"
> > >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> > >> >> "06:1c:78:00:00:7c"
> > >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> > >> "2014-07-23
> > >> >> >> > 08:16:32" \N \N "0" "1"
> > >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> > >> "06:96:4e:00:00:8e"
> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-01-13
> > >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> > >> "2014-01-13
> > >> >> >> > 22:25:44" \N \N "0" "1"
> > >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> > >> >> "06:be:24:00:00:8e"
> > >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> > "2015-02-04
> > >> >> >> 22:54:29"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> > "1"
> > >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> > >> >> "06:cd:d6:00:00:97"
> > >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-09-02
> > >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> > >> "2014-09-02
> > >> >> >> > 09:06:56" \N \N "0" "1"
> > >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> > >> >> "06:6e:0c:00:00:9c"
> > >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> > "2015-02-11
> > >> >> >> 14:32:00"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> > "1"
> > >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> > >> >> "06:b8:b4:00:00:a1"
> > >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-12-02
> > >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> > >> "2014-12-02
> > >> >> >> > 13:58:11" \N \N "0" "1"
> > >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> > >> >> "06:f8:36:00:00:a7"
> > >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-01-21
> > >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> > >> "2015-01-21
> > >> >> >> > 18:46:10" \N \N "0" "1"
> > >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> > >> >> "06:ac:4e:00:00:54"
> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> > >> "2014-07-23
> > >> >> >> > 10:33:59" \N \N "0" "1"
> > >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> > >> >> "06:46:d8:00:00:54"
> > >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2014-07-23
> > >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> > >> "2014-07-23
> > >> >> >> > 14:54:11" \N \N "0" "1"
> > >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> > >> "06:7c:54:00:00:35"
> > >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> > "2015-03-16
> > >> >> >> 12:25:41"
> > >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> > "1"
> > >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> > >> >> "06:c9:e0:00:00:3a"
> > >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2015-03-16
> > >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> > >> "2015-03-16
> > >> >> >> > 12:03:42" \N \N "0" "1"
> > >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> > >> "06:b1:30:00:00:2a"
> > >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "206"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2013-01-11
> > >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> > >> "2013-01-11
> > >> >> >> > 01:11:18" \N \N "0" "1"
> > >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> > >> "06:12:88:00:00:2b"
> > >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "206"
> > >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> > >> "2013-01-11
> > >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> > >> "2013-01-11
> > >> >> >> > 00:35:20" \N \N "0" "1"
> > >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> > >> >> "06:02:a8:00:00:e2"
> > >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> > "vlan://500"
> > >> >> "212"
> > >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> > "2015-02-12
> > >> >> >> 14:16:23"
> > >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> > "1"
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com>
> wrote:
> > >> >> >> >
> > >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> > for
> > >> the
> > >> >> >> >> router code, the null pointer indicates that there's no
> default
> > >> nic
> > >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that
> if
> > >> you
> > >> >> >> >> delete/move just that host you may be able to start the
> router,
> > >> or at
> > >> >> >> >> least get past this. You can also look into the db and see if
> > >> you can
> > >> >> >> >> find its nics in the cloud.nics table and see if there is in
> > >> fact one
> > >> >> >> >> marked default.
> > >> >> >> >>
> > >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <shadowsor@gmail.com
> >
> > >> wrote:
> > >> >> >> >> > Looks like the issue is that null pointer in
> CreateDhcpEntry
> > >> for
> > >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> > after
> > >> >> that.
> > >> >> >> >> > It would suggest some inconsistent/unexpected data when
> > >> creating a
> > >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> > It's
> > >> too
> > >> >> bad
> > >> >> >> >> > that one bad entry is fatal for the whole router.
> > >> >> >> >> >
> > >> >> >> >> >
> > >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> > >> >> >> andrija.panic@gmail.com>
> > >> >> >> >> wrote:
> > >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> > VPS
> > >> >> network
> > >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> > >> >> >> >> >>
> > >> >> >> >> >> MGMT server sends the STOP command to agent, even though
> the
> > >> VM
> > >> >> was
> > >> >> >> >> never
> > >> >> >> >> >> started, BUT the storage provisioning from template to
> > volume
> > >> is
> > >> >> >> done...
> > >> >> >> >> >>
> > >> >> >> >> >> We are also looking into some external help as we speak...
> > >> >> >> >> >>
> > >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> > >> >> >> >> >>
> > >> >> >> >> >>> Hi,
> > >> >> >> >> >>>
> > >> >> >> >> >>> Have you managed to get to the bottom of this?
> > >> >> >> >> >>>
> > >> >> >> >> >>> --
> > >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> > >> >> >> >> >>>
> > >> >> >> >> >>> Nux!
> > >> >> >> >> >>> www.nux.ro
> > >> >> >> >> >>>
> > >> >> >> >> >>> ----- Original Message -----
> > >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> > >> >> >> >> >>> > To: users@cloudstack.apache.org,
> > dev@cloudstack.apache.org
> > >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> > >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> > >> >> >> >> >>>
> > >> >> >> >> >>> > Hi guys,
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> > >> updated
> > >> >> >> right
> > >> >> >> >> >>> before
> > >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> > SHARED
> > >> >> GUEST
> > >> >> >> >> network
> > >> >> >> >> >>> > anymore.
> > >> >> >> >> >>> > And some VMs are down - and cant be started because
> they
> > >> >> depend on
> > >> >> >> >> this
> > >> >> >> >> >>> > VR...
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> > >> fine,
> > >> >> but
> > >> >> >> >> this
> > >> >> >> >> >>> one
> > >> >> >> >> >>> > fro Guest network fails to start:
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > Here you can see, after agent copies template from
> > >> secondary
> > >> >> >> >> storage, to
> > >> >> >> >> >>> > primary local storage, it created base image, and
> backing
> > >> file
> > >> >> -
> > >> >> >> so
> > >> >> >> >> >>> storage
> > >> >> >> >> >>> > setup seems completed.
> > >> >> >> >> >>> >
> > >> >> >> >> >>> > Than all out of sudden we have errors:
> > >> >> >> >> >>>
> > >> >> >> >> >>
> > >> >> >> >> >>
> > >> >> >> >> >>
> > >> >> >> >> >> --
> > >> >> >> >> >>
> > >> >> >> >> >> Andrija Panić
> > >> >> >> >>
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > --
> > >> >> >> >
> > >> >> >> > Andrija Panić
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> >
> > >> >> > Andrija Panić
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrija Panić
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> EOF
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Btw, just in order to avoid the same problem... maybe we could make this
query smarter and if no nic marked as default, but there is only one, then
that single NIC could be returned as default.
On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
> > I will thanks a lot Marcus for hints...
> >
> > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >
> >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> they're just triggered by you trying to fix the situation by
> >> migrating, but the original issue was something else entirely.
> >>
> >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> >> > definitively or my specific setup...
> >> >
> >> > Thanks, I' fixing this now and will let you know.
> >> >
> >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> Yes, each VM should have at least one default nic, so if there's only
> >> >> one nic it should be set to default. Take a db backup first, of
> >> >> course, before messing with it. Any idea how it may have happened? Do
> >> >> you migrate VMs between networks ever?
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> >> change
> >> >> > all of them to default_nic=1... ?
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> VMs can have multiple nics and be on multiple networks. If you
> set a
> >> >> >> nic as default, it becomes the network that the vm has its default
> >> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> >> every VM that might be missing one, and see if that gets your
> router
> >> >> >> up.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hi Marcus,
> >> >> >> >
> >> >> >> > Thanks a lot fot hint
> >> >> >> >
> >> >> >> > True, I have the 0 as the value for some reason in database, for
> >> >> couple
> >> >> >> of
> >> >> >> > NICs....
> >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> broadcast_uri
> >> >> =
> >> >> >> > "vlan://500" and default_nic = 0;
> >> >> >> >
> >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >> >
> >> >> >> > or down there...
> >> >> >> >
> >> >> >> > This Techvee-FileServer server is already running (still not
> dead)
> >> >> and I
> >> >> >> > can see 1 NIC from UI...
> >> >> >> >
> >> >> >> > Should I reset all of these to 1 ?
> >> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >> >
> >> >> >> > vlan://500 in my case limits results only to the network for
> this
> >> VR
> >> >> that
> >> >> >> > is having problems...
> >> >> >> >
> >> >> >> > Any suggestions ?
> >> >> >> >
> >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> >> "gateway"
> >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> >> "isolation_uri"
> >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> >> "ip6_gateway"
> >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> >> "06:70:0a:00:00:ac"
> >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-04
> >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> "2015-02-04
> >> >> >> > 22:06:23" \N \N "0" "1"
> >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> >> "06:fc:c2:00:00:ad"
> >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-03
> >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> "2015-03-03
> >> >> >> > 14:45:47" \N \N "0" "1"
> >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> >> "06:d7:ac:00:00:ae"
> >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-19
> >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> "2015-02-19
> >> >> >> > 02:10:45" \N \N "0" "1"
> >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> "06:b5:72:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> "2014-01-27
> >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> "2014-01-27
> >> >> >> > 13:38:52" \N \N "0" "1"
> >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> >> "06:1d:90:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> "2014-07-23
> >> >> >> > 08:15:18" \N \N "0" "1"
> >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> >> "06:33:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> "2015-03-16
> >> >> >> > 11:41:11" \N \N "0" "1"
> >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> >> "06:8a:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> "2015-03-16
> >> >> >> > 12:02:59" \N \N "0" "1"
> >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> >> "06:c2:c4:00:00:7a"
> >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> "2014-07-23
> >> >> >> > 08:16:12" \N \N "0" "1"
> >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> >> "06:1c:78:00:00:7c"
> >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> "2014-07-23
> >> >> >> > 08:16:32" \N \N "0" "1"
> >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> "06:96:4e:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-01-13
> >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> "2014-01-13
> >> >> >> > 22:25:44" \N \N "0" "1"
> >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> >> "06:be:24:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> >> 22:54:29"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> "1"
> >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> >> "06:cd:d6:00:00:97"
> >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-09-02
> >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> "2014-09-02
> >> >> >> > 09:06:56" \N \N "0" "1"
> >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> >> "06:6e:0c:00:00:9c"
> >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> "2015-02-11
> >> >> >> 14:32:00"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> "1"
> >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> >> "06:b8:b4:00:00:a1"
> >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-12-02
> >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> "2014-12-02
> >> >> >> > 13:58:11" \N \N "0" "1"
> >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> >> "06:f8:36:00:00:a7"
> >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-01-21
> >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> "2015-01-21
> >> >> >> > 18:46:10" \N \N "0" "1"
> >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> >> "06:ac:4e:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> "2014-07-23
> >> >> >> > 10:33:59" \N \N "0" "1"
> >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> >> "06:46:d8:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> "2014-07-23
> >> >> >> > 14:54:11" \N \N "0" "1"
> >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> "06:7c:54:00:00:35"
> >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> "2015-03-16
> >> >> >> 12:25:41"
> >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> "1"
> >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> >> "06:c9:e0:00:00:3a"
> >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> "2015-03-16
> >> >> >> > 12:03:42" \N \N "0" "1"
> >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> "06:b1:30:00:00:2a"
> >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> "2013-01-11
> >> >> >> > 01:11:18" \N \N "0" "1"
> >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> "06:12:88:00:00:2b"
> >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> "2013-01-11
> >> >> >> > 00:35:20" \N \N "0" "1"
> >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> >> "06:02:a8:00:00:e2"
> >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-12
> >> >> >> 14:16:23"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> "1"
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> for
> >> the
> >> >> >> >> router code, the null pointer indicates that there's no default
> >> nic
> >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
> >> you
> >> >> >> >> delete/move just that host you may be able to start the router,
> >> or at
> >> >> >> >> least get past this. You can also look into the db and see if
> >> you can
> >> >> >> >> find its nics in the cloud.nics table and see if there is in
> >> fact one
> >> >> >> >> marked default.
> >> >> >> >>
> >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> >> wrote:
> >> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
> >> for
> >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> after
> >> >> that.
> >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> creating a
> >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> It's
> >> too
> >> >> bad
> >> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> >> andrija.panic@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> VPS
> >> >> network
> >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >> >>
> >> >> >> >> >> MGMT server sends the STOP command to agent, even though the
> >> VM
> >> >> was
> >> >> >> >> never
> >> >> >> >> >> started, BUT the storage provisioning from template to
> volume
> >> is
> >> >> >> done...
> >> >> >> >> >>
> >> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >> >>
> >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi,
> >> >> >> >> >>>
> >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >> >>>
> >> >> >> >> >>> Nux!
> >> >> >> >> >>> www.nux.ro
> >> >> >> >> >>>
> >> >> >> >> >>> ----- Original Message -----
> >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >> >>> > To: users@cloudstack.apache.org,
> dev@cloudstack.apache.org
> >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >> >>>
> >> >> >> >> >>> > Hi guys,
> >> >> >> >> >>> >
> >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> >> updated
> >> >> >> right
> >> >> >> >> >>> before
> >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >> >>> >
> >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> SHARED
> >> >> GUEST
> >> >> >> >> network
> >> >> >> >> >>> > anymore.
> >> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> >> depend on
> >> >> >> >> this
> >> >> >> >> >>> > VR...
> >> >> >> >> >>> >
> >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> >> fine,
> >> >> but
> >> >> >> >> this
> >> >> >> >> >>> one
> >> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >> >>> >
> >> >> >> >> >>> > Here you can see, after agent copies template from
> >> secondary
> >> >> >> >> storage, to
> >> >> >> >> >>> > primary local storage, it created base image, and backing
> >> file
> >> >> -
> >> >> >> so
> >> >> >> >> >>> storage
> >> >> >> >> >>> > setup seems completed.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >>
> >> >> >> >> >> Andrija Panić
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Btw, just in order to avoid the same problem... maybe we could make this
query smarter and if no nic marked as default, but there is only one, then
that single NIC could be returned as default.
On Mon, Mar 16, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
> > I will thanks a lot Marcus for hints...
> >
> > On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> >
> >> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> >> they're just triggered by you trying to fix the situation by
> >> migrating, but the original issue was something else entirely.
> >>
> >> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > I did migrate and also changed accounts, unsucessfully, so some bugs
> >> > definitively or my specific setup...
> >> >
> >> > Thanks, I' fixing this now and will let you know.
> >> >
> >> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> Yes, each VM should have at least one default nic, so if there's only
> >> >> one nic it should be set to default. Take a db backup first, of
> >> >> course, before messing with it. Any idea how it may have happened? Do
> >> >> you migrate VMs between networks ever?
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> >> change
> >> >> > all of them to default_nic=1... ?
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> VMs can have multiple nics and be on multiple networks. If you
> set a
> >> >> >> nic as default, it becomes the network that the vm has its default
> >> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> >> every VM that might be missing one, and see if that gets your
> router
> >> >> >> up.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> >> andrija.panic@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hi Marcus,
> >> >> >> >
> >> >> >> > Thanks a lot fot hint
> >> >> >> >
> >> >> >> > True, I have the 0 as the value for some reason in database, for
> >> >> couple
> >> >> >> of
> >> >> >> > NICs....
> >> >> >> > select * from nics where ip4_address like "46.232%" and
> >> broadcast_uri
> >> >> =
> >> >> >> > "vlan://500" and default_nic = 0;
> >> >> >> >
> >> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >> >
> >> >> >> > or down there...
> >> >> >> >
> >> >> >> > This Techvee-FileServer server is already running (still not
> dead)
> >> >> and I
> >> >> >> > can see 1 NIC from UI...
> >> >> >> >
> >> >> >> > Should I reset all of these to 1 ?
> >> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >> >
> >> >> >> > vlan://500 in my case limits results only to the network for
> this
> >> VR
> >> >> that
> >> >> >> > is having problems...
> >> >> >> >
> >> >> >> > Any suggestions ?
> >> >> >> >
> >> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> >> "gateway"
> >> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> >> "isolation_uri"
> >> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> >> "ip6_gateway"
> >> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> >> "06:70:0a:00:00:ac"
> >> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-04
> >> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> >> "2015-02-04
> >> >> >> > 22:06:23" \N \N "0" "1"
> >> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> >> "06:fc:c2:00:00:ad"
> >> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-03
> >> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> >> "2015-03-03
> >> >> >> > 14:45:47" \N \N "0" "1"
> >> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> >> "06:d7:ac:00:00:ae"
> >> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-02-19
> >> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> >> "2015-02-19
> >> >> >> > 02:10:45" \N \N "0" "1"
> >> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> >> "06:b5:72:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> >> "2014-01-27
> >> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> >> "2014-01-27
> >> >> >> > 13:38:52" \N \N "0" "1"
> >> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> >> "06:1d:90:00:00:72"
> >> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> >> "2014-07-23
> >> >> >> > 08:15:18" \N \N "0" "1"
> >> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> >> "06:33:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> >> "2015-03-16
> >> >> >> > 11:41:11" \N \N "0" "1"
> >> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> >> "06:8a:ac:00:00:77"
> >> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> >> "2015-03-16
> >> >> >> > 12:02:59" \N \N "0" "1"
> >> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> >> "06:c2:c4:00:00:7a"
> >> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> >> "2014-07-23
> >> >> >> > 08:16:12" \N \N "0" "1"
> >> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> >> "06:1c:78:00:00:7c"
> >> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> >> "2014-07-23
> >> >> >> > 08:16:32" \N \N "0" "1"
> >> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> >> "06:96:4e:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-01-13
> >> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> >> "2014-01-13
> >> >> >> > 22:25:44" \N \N "0" "1"
> >> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> >> "06:be:24:00:00:8e"
> >> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> >> 22:54:29"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0"
> "1"
> >> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> >> "06:cd:d6:00:00:97"
> >> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-09-02
> >> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> >> "2014-09-02
> >> >> >> > 09:06:56" \N \N "0" "1"
> >> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> >> "06:6e:0c:00:00:9c"
> >> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0"
> "2015-02-11
> >> >> >> 14:32:00"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0"
> "1"
> >> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> >> "06:b8:b4:00:00:a1"
> >> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-12-02
> >> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> >> "2014-12-02
> >> >> >> > 13:58:11" \N \N "0" "1"
> >> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> >> "06:f8:36:00:00:a7"
> >> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-01-21
> >> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> >> "2015-01-21
> >> >> >> > 18:46:10" \N \N "0" "1"
> >> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> >> "06:ac:4e:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> >> "2014-07-23
> >> >> >> > 10:33:59" \N \N "0" "1"
> >> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> >> "06:46:d8:00:00:54"
> >> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2014-07-23
> >> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> >> "2014-07-23
> >> >> >> > 14:54:11" \N \N "0" "1"
> >> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> >> "06:7c:54:00:00:35"
> >> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2"
> "2015-03-16
> >> >> >> 12:25:41"
> >> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0"
> "1"
> >> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> >> "06:c9:e0:00:00:3a"
> >> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2015-03-16
> >> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> >> "2015-03-16
> >> >> >> > 12:03:42" \N \N "0" "1"
> >> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> >> "06:b1:30:00:00:2a"
> >> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> >> "2013-01-11
> >> >> >> > 01:11:18" \N \N "0" "1"
> >> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> >> "06:12:88:00:00:2b"
> >> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "206"
> >> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> >> "2013-01-11
> >> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> >> "2013-01-11
> >> >> >> > 00:35:20" \N \N "0" "1"
> >> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> >> "06:02:a8:00:00:e2"
> >> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4"
> "vlan://500"
> >> >> "212"
> >> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-12
> >> >> >> 14:16:23"
> >> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0"
> "1"
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree
> for
> >> the
> >> >> >> >> router code, the null pointer indicates that there's no default
> >> nic
> >> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
> >> you
> >> >> >> >> delete/move just that host you may be able to start the router,
> >> or at
> >> >> >> >> least get past this. You can also look into the db and see if
> >> you can
> >> >> >> >> find its nics in the cloud.nics table and see if there is in
> >> fact one
> >> >> >> >> marked default.
> >> >> >> >>
> >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> >> wrote:
> >> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
> >> for
> >> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately
> after
> >> >> that.
> >> >> >> >> > It would suggest some inconsistent/unexpected data when
> >> creating a
> >> >> >> >> > DHCP entry for one of the guests serviced by this router.
> It's
> >> too
> >> >> bad
> >> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> >> andrija.panic@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> Not really - we are painfully migrating stopped VMs, from
> VPS
> >> >> network
> >> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >> >>
> >> >> >> >> >> MGMT server sends the STOP command to agent, even though the
> >> VM
> >> >> was
> >> >> >> >> never
> >> >> >> >> >> started, BUT the storage provisioning from template to
> volume
> >> is
> >> >> >> done...
> >> >> >> >> >>
> >> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >> >>
> >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi,
> >> >> >> >> >>>
> >> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >> >>>
> >> >> >> >> >>> Nux!
> >> >> >> >> >>> www.nux.ro
> >> >> >> >> >>>
> >> >> >> >> >>> ----- Original Message -----
> >> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >> >>> > To: users@cloudstack.apache.org,
> dev@cloudstack.apache.org
> >> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >> >>>
> >> >> >> >> >>> > Hi guys,
> >> >> >> >> >>> >
> >> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> >> updated
> >> >> >> right
> >> >> >> >> >>> before
> >> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >> >>> >
> >> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for
> SHARED
> >> >> GUEST
> >> >> >> >> network
> >> >> >> >> >>> > anymore.
> >> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> >> depend on
> >> >> >> >> this
> >> >> >> >> >>> > VR...
> >> >> >> >> >>> >
> >> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> >> fine,
> >> >> but
> >> >> >> >> this
> >> >> >> >> >>> one
> >> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >> >>> >
> >> >> >> >> >>> > Here you can see, after agent copies template from
> >> secondary
> >> >> >> >> storage, to
> >> >> >> >> >>> > primary local storage, it created base image, and backing
> >> file
> >> >> -
> >> >> >> so
> >> >> >> >> >>> storage
> >> >> >> >> >>> > setup seems completed.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >>
> >> >> >> >> >> Andrija Panić
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Good to hear. Glad I could lend another pair of eyes.
On Mon, Mar 16, 2015 at 9:19 AM, Andrija Panic <an...@gmail.com> wrote:
> FIXED !!!
>
> Thanks a lot Marcus, this is the second time you saved me from the deep
> $$it..
>
> Only 1 VM that had only 1 NIC and not set to default in DB - so just after
> changing that default_nic=1, destroyed the VR, and new one was recreated.
>
> Thanks a lot for help !
>
>
> On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
>
>> I will thanks a lot Marcus for hints...
>>
>> On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>>
>>> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>>> they're just triggered by you trying to fix the situation by
>>> migrating, but the original issue was something else entirely.
>>>
>>> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
>>> wrote:
>>> > I did migrate and also changed accounts, unsucessfully, so some bugs
>>> > definitively or my specific setup...
>>> >
>>> > Thanks, I' fixing this now and will let you know.
>>> >
>>> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>>> >
>>> >> Yes, each VM should have at least one default nic, so if there's only
>>> >> one nic it should be set to default. Take a db backup first, of
>>> >> course, before messing with it. Any idea how it may have happened? Do
>>> >> you migrate VMs between networks ever?
>>> >>
>>> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>>> andrija.panic@gmail.com>
>>> >> wrote:
>>> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
>>> change
>>> >> > all of them to default_nic=1... ?
>>> >> >
>>> >> >
>>> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>>> >> >
>>> >> >> VMs can have multiple nics and be on multiple networks. If you set a
>>> >> >> nic as default, it becomes the network that the vm has its default
>>> >> >> route on. Every VM should have a default nic, and if it doesn't I
>>> >> >> wonder how it might have happened (maybe a specific combination of
>>> >> >> add/delete nic triggered a bug?). You should set a default nic for
>>> >> >> every VM that might be missing one, and see if that gets your router
>>> >> >> up.
>>> >> >>
>>> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>>> andrija.panic@gmail.com
>>> >> >
>>> >> >> wrote:
>>> >> >> > Hi Marcus,
>>> >> >> >
>>> >> >> > Thanks a lot fot hint
>>> >> >> >
>>> >> >> > True, I have the 0 as the value for some reason in database, for
>>> >> couple
>>> >> >> of
>>> >> >> > NICs....
>>> >> >> > select * from nics where ip4_address like "46.232%" and
>>> broadcast_uri
>>> >> =
>>> >> >> > "vlan://500" and default_nic = 0;
>>> >> >> >
>>> >> >> > results: http://pastebin.com/rDAe2RY9
>>> >> >> >
>>> >> >> > or down there...
>>> >> >> >
>>> >> >> > This Techvee-FileServer server is already running (still not dead)
>>> >> and I
>>> >> >> > can see 1 NIC from UI...
>>> >> >> >
>>> >> >> > Should I reset all of these to 1 ?
>>> >> >> > What is the purpose of this field default_nic = 0.
>>> >> >> >
>>> >> >> > vlan://500 in my case limits results only to the network for this
>>> VR
>>> >> that
>>> >> >> > is having problems...
>>> >> >> >
>>> >> >> > Any suggestions ?
>>> >> >> >
>>> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>>> >> "gateway"
>>> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>>> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>>> >> >> "isolation_uri"
>>> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>>> >> "ip6_gateway"
>>> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>>> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>>> >> "06:70:0a:00:00:ac"
>>> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-02-04
>>> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>>> "2015-02-04
>>> >> >> > 22:06:23" \N \N "0" "1"
>>> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>>> >> "06:fc:c2:00:00:ad"
>>> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-03
>>> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>>> "2015-03-03
>>> >> >> > 14:45:47" \N \N "0" "1"
>>> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>>> >> "06:d7:ac:00:00:ae"
>>> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-02-19
>>> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>>> "2015-02-19
>>> >> >> > 02:10:45" \N \N "0" "1"
>>> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>>> "06:b5:72:00:00:72"
>>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>>> "2014-01-27
>>> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>>> "2014-01-27
>>> >> >> > 13:38:52" \N \N "0" "1"
>>> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>>> >> "06:1d:90:00:00:72"
>>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>>> "2014-07-23
>>> >> >> > 08:15:18" \N \N "0" "1"
>>> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>>> >> "06:33:ac:00:00:77"
>>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>>> "2015-03-16
>>> >> >> > 11:41:11" \N \N "0" "1"
>>> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>>> >> "06:8a:ac:00:00:77"
>>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>>> "2015-03-16
>>> >> >> > 12:02:59" \N \N "0" "1"
>>> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>>> >> "06:c2:c4:00:00:7a"
>>> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>>> "2014-07-23
>>> >> >> > 08:16:12" \N \N "0" "1"
>>> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>>> >> "06:1c:78:00:00:7c"
>>> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>>> "2014-07-23
>>> >> >> > 08:16:32" \N \N "0" "1"
>>> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>>> "06:96:4e:00:00:8e"
>>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-01-13
>>> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>>> "2014-01-13
>>> >> >> > 22:25:44" \N \N "0" "1"
>>> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>>> >> "06:be:24:00:00:8e"
>>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>>> >> >> 22:54:29"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>>> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>>> >> "06:cd:d6:00:00:97"
>>> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-09-02
>>> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>>> "2014-09-02
>>> >> >> > 09:06:56" \N \N "0" "1"
>>> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>>> >> "06:6e:0c:00:00:9c"
>>> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>>> >> >> 14:32:00"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>>> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>>> >> "06:b8:b4:00:00:a1"
>>> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-12-02
>>> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>>> "2014-12-02
>>> >> >> > 13:58:11" \N \N "0" "1"
>>> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>>> >> "06:f8:36:00:00:a7"
>>> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-01-21
>>> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>>> "2015-01-21
>>> >> >> > 18:46:10" \N \N "0" "1"
>>> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>>> >> "06:ac:4e:00:00:54"
>>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>>> "2014-07-23
>>> >> >> > 10:33:59" \N \N "0" "1"
>>> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>>> >> "06:46:d8:00:00:54"
>>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2014-07-23
>>> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>>> "2014-07-23
>>> >> >> > 14:54:11" \N \N "0" "1"
>>> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>>> "06:7c:54:00:00:35"
>>> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>>> >> >> 12:25:41"
>>> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>>> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>>> >> "06:c9:e0:00:00:3a"
>>> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2015-03-16
>>> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>>> "2015-03-16
>>> >> >> > 12:03:42" \N \N "0" "1"
>>> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>>> "06:b1:30:00:00:2a"
>>> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "206"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2013-01-11
>>> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>>> "2013-01-11
>>> >> >> > 01:11:18" \N \N "0" "1"
>>> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>>> "06:12:88:00:00:2b"
>>> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "206"
>>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>>> "2013-01-11
>>> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>>> "2013-01-11
>>> >> >> > 00:35:20" \N \N "0" "1"
>>> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>>> >> "06:02:a8:00:00:e2"
>>> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>>> >> "212"
>>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>>> >> >> 14:16:23"
>>> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>>> >> >> >
>>> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
>>> the
>>> >> >> >> router code, the null pointer indicates that there's no default
>>> nic
>>> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
>>> you
>>> >> >> >> delete/move just that host you may be able to start the router,
>>> or at
>>> >> >> >> least get past this. You can also look into the db and see if
>>> you can
>>> >> >> >> find its nics in the cloud.nics table and see if there is in
>>> fact one
>>> >> >> >> marked default.
>>> >> >> >>
>>> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
>>> wrote:
>>> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
>>> for
>>> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>>> >> that.
>>> >> >> >> > It would suggest some inconsistent/unexpected data when
>>> creating a
>>> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
>>> too
>>> >> bad
>>> >> >> >> > that one bad entry is fatal for the whole router.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>>> >> >> andrija.panic@gmail.com>
>>> >> >> >> wrote:
>>> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>>> >> network
>>> >> >> >> >> (Guest Shared netwotk) to VPCs...
>>> >> >> >> >>
>>> >> >> >> >> MGMT server sends the STOP command to agent, even though the
>>> VM
>>> >> was
>>> >> >> >> never
>>> >> >> >> >> started, BUT the storage provisioning from template to volume
>>> is
>>> >> >> done...
>>> >> >> >> >>
>>> >> >> >> >> We are also looking into some external help as we speak...
>>> >> >> >> >>
>>> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>>> >> >> >> >>
>>> >> >> >> >>> Hi,
>>> >> >> >> >>>
>>> >> >> >> >>> Have you managed to get to the bottom of this?
>>> >> >> >> >>>
>>> >> >> >> >>> --
>>> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>>> >> >> >> >>>
>>> >> >> >> >>> Nux!
>>> >> >> >> >>> www.nux.ro
>>> >> >> >> >>>
>>> >> >> >> >>> ----- Original Message -----
>>> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>>> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>>> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>>> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>>> >> >> >> >>>
>>> >> >> >> >>> > Hi guys,
>>> >> >> >> >>> >
>>> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>>> updated
>>> >> >> right
>>> >> >> >> >>> before
>>> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>>> >> >> >> >>> >
>>> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>>> >> GUEST
>>> >> >> >> network
>>> >> >> >> >>> > anymore.
>>> >> >> >> >>> > And some VMs are down - and cant be started because they
>>> >> depend on
>>> >> >> >> this
>>> >> >> >> >>> > VR...
>>> >> >> >> >>> >
>>> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>>> fine,
>>> >> but
>>> >> >> >> this
>>> >> >> >> >>> one
>>> >> >> >> >>> > fro Guest network fails to start:
>>> >> >> >> >>> >
>>> >> >> >> >>> > Here you can see, after agent copies template from
>>> secondary
>>> >> >> >> storage, to
>>> >> >> >> >>> > primary local storage, it created base image, and backing
>>> file
>>> >> -
>>> >> >> so
>>> >> >> >> >>> storage
>>> >> >> >> >>> > setup seems completed.
>>> >> >> >> >>> >
>>> >> >> >> >>> > Than all out of sudden we have errors:
>>> >> >> >> >>>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> --
>>> >> >> >> >>
>>> >> >> >> >> Andrija Panić
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> >
>>> >> >> > Andrija Panić
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Andrija Panić
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
FIXED !!!
Thanks a lot Marcus, this is the second time you saved me from the deep
$$it..
Only 1 VM that had only 1 NIC and not set to default in DB - so just after
changing that default_nic=1, destroyed the VR, and new one was recreated.
Thanks a lot for help !
On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
> I will thanks a lot Marcus for hints...
>
> On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>
>> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>> they're just triggered by you trying to fix the situation by
>> migrating, but the original issue was something else entirely.
>>
>> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > I did migrate and also changed accounts, unsucessfully, so some bugs
>> > definitively or my specific setup...
>> >
>> > Thanks, I' fixing this now and will let you know.
>> >
>> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>> >
>> >> Yes, each VM should have at least one default nic, so if there's only
>> >> one nic it should be set to default. Take a db backup first, of
>> >> course, before messing with it. Any idea how it may have happened? Do
>> >> you migrate VMs between networks ever?
>> >>
>> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>> andrija.panic@gmail.com>
>> >> wrote:
>> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
>> change
>> >> > all of them to default_nic=1... ?
>> >> >
>> >> >
>> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> >> >
>> >> >> VMs can have multiple nics and be on multiple networks. If you set a
>> >> >> nic as default, it becomes the network that the vm has its default
>> >> >> route on. Every VM should have a default nic, and if it doesn't I
>> >> >> wonder how it might have happened (maybe a specific combination of
>> >> >> add/delete nic triggered a bug?). You should set a default nic for
>> >> >> every VM that might be missing one, and see if that gets your router
>> >> >> up.
>> >> >>
>> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>> andrija.panic@gmail.com
>> >> >
>> >> >> wrote:
>> >> >> > Hi Marcus,
>> >> >> >
>> >> >> > Thanks a lot fot hint
>> >> >> >
>> >> >> > True, I have the 0 as the value for some reason in database, for
>> >> couple
>> >> >> of
>> >> >> > NICs....
>> >> >> > select * from nics where ip4_address like "46.232%" and
>> broadcast_uri
>> >> =
>> >> >> > "vlan://500" and default_nic = 0;
>> >> >> >
>> >> >> > results: http://pastebin.com/rDAe2RY9
>> >> >> >
>> >> >> > or down there...
>> >> >> >
>> >> >> > This Techvee-FileServer server is already running (still not dead)
>> >> and I
>> >> >> > can see 1 NIC from UI...
>> >> >> >
>> >> >> > Should I reset all of these to 1 ?
>> >> >> > What is the purpose of this field default_nic = 0.
>> >> >> >
>> >> >> > vlan://500 in my case limits results only to the network for this
>> VR
>> >> that
>> >> >> > is having problems...
>> >> >> >
>> >> >> > Any suggestions ?
>> >> >> >
>> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>> >> "gateway"
>> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> >> >> "isolation_uri"
>> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> >> "ip6_gateway"
>> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> >> "06:70:0a:00:00:ac"
>> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-02-04
>> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>> "2015-02-04
>> >> >> > 22:06:23" \N \N "0" "1"
>> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> >> "06:fc:c2:00:00:ad"
>> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-03
>> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>> "2015-03-03
>> >> >> > 14:45:47" \N \N "0" "1"
>> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> >> "06:d7:ac:00:00:ae"
>> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-02-19
>> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>> "2015-02-19
>> >> >> > 02:10:45" \N \N "0" "1"
>> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>> "06:b5:72:00:00:72"
>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>> "2014-01-27
>> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>> "2014-01-27
>> >> >> > 13:38:52" \N \N "0" "1"
>> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> >> "06:1d:90:00:00:72"
>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>> "2014-07-23
>> >> >> > 08:15:18" \N \N "0" "1"
>> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> >> "06:33:ac:00:00:77"
>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>> "2015-03-16
>> >> >> > 11:41:11" \N \N "0" "1"
>> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> >> "06:8a:ac:00:00:77"
>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>> "2015-03-16
>> >> >> > 12:02:59" \N \N "0" "1"
>> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> >> "06:c2:c4:00:00:7a"
>> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>> "2014-07-23
>> >> >> > 08:16:12" \N \N "0" "1"
>> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> >> "06:1c:78:00:00:7c"
>> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>> "2014-07-23
>> >> >> > 08:16:32" \N \N "0" "1"
>> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>> "06:96:4e:00:00:8e"
>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-01-13
>> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>> "2014-01-13
>> >> >> > 22:25:44" \N \N "0" "1"
>> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> >> "06:be:24:00:00:8e"
>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> >> 22:54:29"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> >> "06:cd:d6:00:00:97"
>> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-09-02
>> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>> "2014-09-02
>> >> >> > 09:06:56" \N \N "0" "1"
>> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> >> "06:6e:0c:00:00:9c"
>> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> >> >> 14:32:00"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> >> "06:b8:b4:00:00:a1"
>> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-12-02
>> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>> "2014-12-02
>> >> >> > 13:58:11" \N \N "0" "1"
>> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> >> "06:f8:36:00:00:a7"
>> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-01-21
>> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>> "2015-01-21
>> >> >> > 18:46:10" \N \N "0" "1"
>> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> >> "06:ac:4e:00:00:54"
>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>> "2014-07-23
>> >> >> > 10:33:59" \N \N "0" "1"
>> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> >> "06:46:d8:00:00:54"
>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>> "2014-07-23
>> >> >> > 14:54:11" \N \N "0" "1"
>> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>> "06:7c:54:00:00:35"
>> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> >> >> 12:25:41"
>> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> >> "06:c9:e0:00:00:3a"
>> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>> "2015-03-16
>> >> >> > 12:03:42" \N \N "0" "1"
>> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>> "06:b1:30:00:00:2a"
>> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "206"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2013-01-11
>> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>> "2013-01-11
>> >> >> > 01:11:18" \N \N "0" "1"
>> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>> "06:12:88:00:00:2b"
>> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "206"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2013-01-11
>> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>> "2013-01-11
>> >> >> > 00:35:20" \N \N "0" "1"
>> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> >> "06:02:a8:00:00:e2"
>> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> >> >> 14:16:23"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >> >> >
>> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
>> the
>> >> >> >> router code, the null pointer indicates that there's no default
>> nic
>> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
>> you
>> >> >> >> delete/move just that host you may be able to start the router,
>> or at
>> >> >> >> least get past this. You can also look into the db and see if
>> you can
>> >> >> >> find its nics in the cloud.nics table and see if there is in
>> fact one
>> >> >> >> marked default.
>> >> >> >>
>> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
>> wrote:
>> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
>> for
>> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>> >> that.
>> >> >> >> > It would suggest some inconsistent/unexpected data when
>> creating a
>> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
>> too
>> >> bad
>> >> >> >> > that one bad entry is fatal for the whole router.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> >> >> andrija.panic@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>> >> network
>> >> >> >> >> (Guest Shared netwotk) to VPCs...
>> >> >> >> >>
>> >> >> >> >> MGMT server sends the STOP command to agent, even though the
>> VM
>> >> was
>> >> >> >> never
>> >> >> >> >> started, BUT the storage provisioning from template to volume
>> is
>> >> >> done...
>> >> >> >> >>
>> >> >> >> >> We are also looking into some external help as we speak...
>> >> >> >> >>
>> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >> >> >>
>> >> >> >> >>> Hi,
>> >> >> >> >>>
>> >> >> >> >>> Have you managed to get to the bottom of this?
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >> >> >>>
>> >> >> >> >>> Nux!
>> >> >> >> >>> www.nux.ro
>> >> >> >> >>>
>> >> >> >> >>> ----- Original Message -----
>> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >> >> >>>
>> >> >> >> >>> > Hi guys,
>> >> >> >> >>> >
>> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>> updated
>> >> >> right
>> >> >> >> >>> before
>> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >> >> >>> >
>> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>> >> GUEST
>> >> >> >> network
>> >> >> >> >>> > anymore.
>> >> >> >> >>> > And some VMs are down - and cant be started because they
>> >> depend on
>> >> >> >> this
>> >> >> >> >>> > VR...
>> >> >> >> >>> >
>> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>> fine,
>> >> but
>> >> >> >> this
>> >> >> >> >>> one
>> >> >> >> >>> > fro Guest network fails to start:
>> >> >> >> >>> >
>> >> >> >> >>> > Here you can see, after agent copies template from
>> secondary
>> >> >> >> storage, to
>> >> >> >> >>> > primary local storage, it created base image, and backing
>> file
>> >> -
>> >> >> so
>> >> >> >> >>> storage
>> >> >> >> >>> > setup seems completed.
>> >> >> >> >>> >
>> >> >> >> >>> > Than all out of sudden we have errors:
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >>
>> >> >> >> >> Andrija Panić
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Andrija Panić
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
FIXED !!!
Thanks a lot Marcus, this is the second time you saved me from the deep
$$it..
Only 1 VM that had only 1 NIC and not set to default in DB - so just after
changing that default_nic=1, destroyed the VR, and new one was recreated.
Thanks a lot for help !
On 16 March 2015 at 16:52, Andrija Panic <an...@gmail.com> wrote:
> I will thanks a lot Marcus for hints...
>
> On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
>
>> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
>> they're just triggered by you trying to fix the situation by
>> migrating, but the original issue was something else entirely.
>>
>> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > I did migrate and also changed accounts, unsucessfully, so some bugs
>> > definitively or my specific setup...
>> >
>> > Thanks, I' fixing this now and will let you know.
>> >
>> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>> >
>> >> Yes, each VM should have at least one default nic, so if there's only
>> >> one nic it should be set to default. Take a db backup first, of
>> >> course, before messing with it. Any idea how it may have happened? Do
>> >> you migrate VMs between networks ever?
>> >>
>> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <
>> andrija.panic@gmail.com>
>> >> wrote:
>> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
>> change
>> >> > all of them to default_nic=1... ?
>> >> >
>> >> >
>> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> >> >
>> >> >> VMs can have multiple nics and be on multiple networks. If you set a
>> >> >> nic as default, it becomes the network that the vm has its default
>> >> >> route on. Every VM should have a default nic, and if it doesn't I
>> >> >> wonder how it might have happened (maybe a specific combination of
>> >> >> add/delete nic triggered a bug?). You should set a default nic for
>> >> >> every VM that might be missing one, and see if that gets your router
>> >> >> up.
>> >> >>
>> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
>> andrija.panic@gmail.com
>> >> >
>> >> >> wrote:
>> >> >> > Hi Marcus,
>> >> >> >
>> >> >> > Thanks a lot fot hint
>> >> >> >
>> >> >> > True, I have the 0 as the value for some reason in database, for
>> >> couple
>> >> >> of
>> >> >> > NICs....
>> >> >> > select * from nics where ip4_address like "46.232%" and
>> broadcast_uri
>> >> =
>> >> >> > "vlan://500" and default_nic = 0;
>> >> >> >
>> >> >> > results: http://pastebin.com/rDAe2RY9
>> >> >> >
>> >> >> > or down there...
>> >> >> >
>> >> >> > This Techvee-FileServer server is already running (still not dead)
>> >> and I
>> >> >> > can see 1 NIC from UI...
>> >> >> >
>> >> >> > Should I reset all of these to 1 ?
>> >> >> > What is the purpose of this field default_nic = 0.
>> >> >> >
>> >> >> > vlan://500 in my case limits results only to the network for this
>> VR
>> >> that
>> >> >> > is having problems...
>> >> >> >
>> >> >> > Any suggestions ?
>> >> >> >
>> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>> >> "gateway"
>> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> >> >> "isolation_uri"
>> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> >> "ip6_gateway"
>> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> >> "06:70:0a:00:00:ac"
>> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-02-04
>> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
>> "2015-02-04
>> >> >> > 22:06:23" \N \N "0" "1"
>> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> >> "06:fc:c2:00:00:ad"
>> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-03
>> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
>> "2015-03-03
>> >> >> > 14:45:47" \N \N "0" "1"
>> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> >> "06:d7:ac:00:00:ae"
>> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-02-19
>> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
>> "2015-02-19
>> >> >> > 02:10:45" \N \N "0" "1"
>> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
>> "06:b5:72:00:00:72"
>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
>> "2014-01-27
>> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
>> "2014-01-27
>> >> >> > 13:38:52" \N \N "0" "1"
>> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> >> "06:1d:90:00:00:72"
>> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
>> "2014-07-23
>> >> >> > 08:15:18" \N \N "0" "1"
>> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> >> "06:33:ac:00:00:77"
>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
>> "2015-03-16
>> >> >> > 11:41:11" \N \N "0" "1"
>> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> >> "06:8a:ac:00:00:77"
>> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
>> "2015-03-16
>> >> >> > 12:02:59" \N \N "0" "1"
>> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> >> "06:c2:c4:00:00:7a"
>> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
>> "2014-07-23
>> >> >> > 08:16:12" \N \N "0" "1"
>> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> >> "06:1c:78:00:00:7c"
>> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
>> "2014-07-23
>> >> >> > 08:16:32" \N \N "0" "1"
>> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
>> "06:96:4e:00:00:8e"
>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-01-13
>> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
>> "2014-01-13
>> >> >> > 22:25:44" \N \N "0" "1"
>> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> >> "06:be:24:00:00:8e"
>> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> >> 22:54:29"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> >> "06:cd:d6:00:00:97"
>> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-09-02
>> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
>> "2014-09-02
>> >> >> > 09:06:56" \N \N "0" "1"
>> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> >> "06:6e:0c:00:00:9c"
>> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> >> >> 14:32:00"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> >> "06:b8:b4:00:00:a1"
>> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-12-02
>> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
>> "2014-12-02
>> >> >> > 13:58:11" \N \N "0" "1"
>> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> >> "06:f8:36:00:00:a7"
>> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-01-21
>> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
>> "2015-01-21
>> >> >> > 18:46:10" \N \N "0" "1"
>> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> >> "06:ac:4e:00:00:54"
>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
>> "2014-07-23
>> >> >> > 10:33:59" \N \N "0" "1"
>> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> >> "06:46:d8:00:00:54"
>> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2014-07-23
>> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
>> "2014-07-23
>> >> >> > 14:54:11" \N \N "0" "1"
>> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
>> "06:7c:54:00:00:35"
>> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> >> >> 12:25:41"
>> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> >> "06:c9:e0:00:00:3a"
>> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2015-03-16
>> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
>> "2015-03-16
>> >> >> > 12:03:42" \N \N "0" "1"
>> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
>> "06:b1:30:00:00:2a"
>> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "206"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2013-01-11
>> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
>> "2013-01-11
>> >> >> > 01:11:18" \N \N "0" "1"
>> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
>> "06:12:88:00:00:2b"
>> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "206"
>> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
>> "2013-01-11
>> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
>> "2013-01-11
>> >> >> > 00:35:20" \N \N "0" "1"
>> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> >> "06:02:a8:00:00:e2"
>> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> >> "212"
>> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> >> >> 14:16:23"
>> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >> >> >
>> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
>> the
>> >> >> >> router code, the null pointer indicates that there's no default
>> nic
>> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if
>> you
>> >> >> >> delete/move just that host you may be able to start the router,
>> or at
>> >> >> >> least get past this. You can also look into the db and see if
>> you can
>> >> >> >> find its nics in the cloud.nics table and see if there is in
>> fact one
>> >> >> >> marked default.
>> >> >> >>
>> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
>> wrote:
>> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry
>> for
>> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>> >> that.
>> >> >> >> > It would suggest some inconsistent/unexpected data when
>> creating a
>> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
>> too
>> >> bad
>> >> >> >> > that one bad entry is fatal for the whole router.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> >> >> andrija.panic@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>> >> network
>> >> >> >> >> (Guest Shared netwotk) to VPCs...
>> >> >> >> >>
>> >> >> >> >> MGMT server sends the STOP command to agent, even though the
>> VM
>> >> was
>> >> >> >> never
>> >> >> >> >> started, BUT the storage provisioning from template to volume
>> is
>> >> >> done...
>> >> >> >> >>
>> >> >> >> >> We are also looking into some external help as we speak...
>> >> >> >> >>
>> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >> >> >>
>> >> >> >> >>> Hi,
>> >> >> >> >>>
>> >> >> >> >>> Have you managed to get to the bottom of this?
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >> >> >>>
>> >> >> >> >>> Nux!
>> >> >> >> >>> www.nux.ro
>> >> >> >> >>>
>> >> >> >> >>> ----- Original Message -----
>> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >> >> >>>
>> >> >> >> >>> > Hi guys,
>> >> >> >> >>> >
>> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
>> updated
>> >> >> right
>> >> >> >> >>> before
>> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >> >> >>> >
>> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>> >> GUEST
>> >> >> >> network
>> >> >> >> >>> > anymore.
>> >> >> >> >>> > And some VMs are down - and cant be started because they
>> >> depend on
>> >> >> >> this
>> >> >> >> >>> > VR...
>> >> >> >> >>> >
>> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
>> fine,
>> >> but
>> >> >> >> this
>> >> >> >> >>> one
>> >> >> >> >>> > fro Guest network fails to start:
>> >> >> >> >>> >
>> >> >> >> >>> > Here you can see, after agent copies template from
>> secondary
>> >> >> >> storage, to
>> >> >> >> >>> > primary local storage, it created base image, and backing
>> file
>> >> -
>> >> >> so
>> >> >> >> >>> storage
>> >> >> >> >>> > setup seems completed.
>> >> >> >> >>> >
>> >> >> >> >>> > Than all out of sudden we have errors:
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >>
>> >> >> >> >> Andrija Panić
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Andrija Panić
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I will thanks a lot Marcus for hints...
On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> they're just triggered by you trying to fix the situation by
> migrating, but the original issue was something else entirely.
>
> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > I did migrate and also changed accounts, unsucessfully, so some bugs
> > definitively or my specific setup...
> >
> > Thanks, I' fixing this now and will let you know.
> >
> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >
> >> Yes, each VM should have at least one default nic, so if there's only
> >> one nic it should be set to default. Take a db backup first, of
> >> course, before messing with it. Any idea how it may have happened? Do
> >> you migrate VMs between networks ever?
> >>
> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> change
> >> > all of them to default_nic=1... ?
> >> >
> >> >
> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> VMs can have multiple nics and be on multiple networks. If you set a
> >> >> nic as default, it becomes the network that the vm has its default
> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> every VM that might be missing one, and see if that gets your router
> >> >> up.
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> andrija.panic@gmail.com
> >> >
> >> >> wrote:
> >> >> > Hi Marcus,
> >> >> >
> >> >> > Thanks a lot fot hint
> >> >> >
> >> >> > True, I have the 0 as the value for some reason in database, for
> >> couple
> >> >> of
> >> >> > NICs....
> >> >> > select * from nics where ip4_address like "46.232%" and
> broadcast_uri
> >> =
> >> >> > "vlan://500" and default_nic = 0;
> >> >> >
> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >
> >> >> > or down there...
> >> >> >
> >> >> > This Techvee-FileServer server is already running (still not dead)
> >> and I
> >> >> > can see 1 NIC from UI...
> >> >> >
> >> >> > Should I reset all of these to 1 ?
> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >
> >> >> > vlan://500 in my case limits results only to the network for this
> VR
> >> that
> >> >> > is having problems...
> >> >> >
> >> >> > Any suggestions ?
> >> >> >
> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> "gateway"
> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> "isolation_uri"
> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> "ip6_gateway"
> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> "06:70:0a:00:00:ac"
> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> "2015-02-04
> >> >> > 22:06:23" \N \N "0" "1"
> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> "06:fc:c2:00:00:ad"
> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-03
> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> "2015-03-03
> >> >> > 14:45:47" \N \N "0" "1"
> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> "06:d7:ac:00:00:ae"
> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-19
> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> "2015-02-19
> >> >> > 02:10:45" \N \N "0" "1"
> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> "06:b5:72:00:00:72"
> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> "2014-01-27
> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> "2014-01-27
> >> >> > 13:38:52" \N \N "0" "1"
> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> "06:1d:90:00:00:72"
> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> "2014-07-23
> >> >> > 08:15:18" \N \N "0" "1"
> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> "06:33:ac:00:00:77"
> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> "2015-03-16
> >> >> > 11:41:11" \N \N "0" "1"
> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> "06:8a:ac:00:00:77"
> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> "2015-03-16
> >> >> > 12:02:59" \N \N "0" "1"
> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> "06:c2:c4:00:00:7a"
> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> "2014-07-23
> >> >> > 08:16:12" \N \N "0" "1"
> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> "06:1c:78:00:00:7c"
> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> "2014-07-23
> >> >> > 08:16:32" \N \N "0" "1"
> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> "06:96:4e:00:00:8e"
> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-01-13
> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> "2014-01-13
> >> >> > 22:25:44" \N \N "0" "1"
> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> "06:be:24:00:00:8e"
> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> >> 22:54:29"
> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> "06:cd:d6:00:00:97"
> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-09-02
> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> "2014-09-02
> >> >> > 09:06:56" \N \N "0" "1"
> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> "06:6e:0c:00:00:9c"
> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> >> >> 14:32:00"
> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> "06:b8:b4:00:00:a1"
> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-12-02
> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> "2014-12-02
> >> >> > 13:58:11" \N \N "0" "1"
> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> "06:f8:36:00:00:a7"
> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-01-21
> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> "2015-01-21
> >> >> > 18:46:10" \N \N "0" "1"
> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> "06:ac:4e:00:00:54"
> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> "2014-07-23
> >> >> > 10:33:59" \N \N "0" "1"
> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> "06:46:d8:00:00:54"
> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> "2014-07-23
> >> >> > 14:54:11" \N \N "0" "1"
> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> "06:7c:54:00:00:35"
> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> >> >> 12:25:41"
> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> "06:c9:e0:00:00:3a"
> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> "2015-03-16
> >> >> > 12:03:42" \N \N "0" "1"
> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> "06:b1:30:00:00:2a"
> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "206"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2013-01-11
> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> "2013-01-11
> >> >> > 01:11:18" \N \N "0" "1"
> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> "06:12:88:00:00:2b"
> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "206"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2013-01-11
> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> "2013-01-11
> >> >> > 00:35:20" \N \N "0" "1"
> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> "06:02:a8:00:00:e2"
> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> >> >> 14:16:23"
> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
> the
> >> >> >> router code, the null pointer indicates that there's no default
> nic
> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> >> >> delete/move just that host you may be able to start the router,
> or at
> >> >> >> least get past this. You can also look into the db and see if you
> can
> >> >> >> find its nics in the cloud.nics table and see if there is in fact
> one
> >> >> >> marked default.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> wrote:
> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
> >> that.
> >> >> >> > It would suggest some inconsistent/unexpected data when
> creating a
> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
> too
> >> bad
> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >
> >> >> >> >
> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> andrija.panic@gmail.com>
> >> >> >> wrote:
> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
> >> network
> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >>
> >> >> >> >> MGMT server sends the STOP command to agent, even though the VM
> >> was
> >> >> >> never
> >> >> >> >> started, BUT the storage provisioning from template to volume
> is
> >> >> done...
> >> >> >> >>
> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >>
> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >>
> >> >> >> >>> Hi,
> >> >> >> >>>
> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >>>
> >> >> >> >>> Nux!
> >> >> >> >>> www.nux.ro
> >> >> >> >>>
> >> >> >> >>> ----- Original Message -----
> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >>>
> >> >> >> >>> > Hi guys,
> >> >> >> >>> >
> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> updated
> >> >> right
> >> >> >> >>> before
> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >>> >
> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
> >> GUEST
> >> >> >> network
> >> >> >> >>> > anymore.
> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> depend on
> >> >> >> this
> >> >> >> >>> > VR...
> >> >> >> >>> >
> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> fine,
> >> but
> >> >> >> this
> >> >> >> >>> one
> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >>> >
> >> >> >> >>> > Here you can see, after agent copies template from secondary
> >> >> >> storage, to
> >> >> >> >>> > primary local storage, it created base image, and backing
> file
> >> -
> >> >> so
> >> >> >> >>> storage
> >> >> >> >>> > setup seems completed.
> >> >> >> >>> >
> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >>
> >> >> >> >> Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I will thanks a lot Marcus for hints...
On 16 March 2015 at 16:49, Marcus <sh...@gmail.com> wrote:
> Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
> they're just triggered by you trying to fix the situation by
> migrating, but the original issue was something else entirely.
>
> On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > I did migrate and also changed accounts, unsucessfully, so some bugs
> > definitively or my specific setup...
> >
> > Thanks, I' fixing this now and will let you know.
> >
> > On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> >
> >> Yes, each VM should have at least one default nic, so if there's only
> >> one nic it should be set to default. Take a db backup first, of
> >> course, before messing with it. Any idea how it may have happened? Do
> >> you migrate VMs between networks ever?
> >>
> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to
> change
> >> > all of them to default_nic=1... ?
> >> >
> >> >
> >> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> VMs can have multiple nics and be on multiple networks. If you set a
> >> >> nic as default, it becomes the network that the vm has its default
> >> >> route on. Every VM should have a default nic, and if it doesn't I
> >> >> wonder how it might have happened (maybe a specific combination of
> >> >> add/delete nic triggered a bug?). You should set a default nic for
> >> >> every VM that might be missing one, and see if that gets your router
> >> >> up.
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <
> andrija.panic@gmail.com
> >> >
> >> >> wrote:
> >> >> > Hi Marcus,
> >> >> >
> >> >> > Thanks a lot fot hint
> >> >> >
> >> >> > True, I have the 0 as the value for some reason in database, for
> >> couple
> >> >> of
> >> >> > NICs....
> >> >> > select * from nics where ip4_address like "46.232%" and
> broadcast_uri
> >> =
> >> >> > "vlan://500" and default_nic = 0;
> >> >> >
> >> >> > results: http://pastebin.com/rDAe2RY9
> >> >> >
> >> >> > or down there...
> >> >> >
> >> >> > This Techvee-FileServer server is already running (still not dead)
> >> and I
> >> >> > can see 1 NIC from UI...
> >> >> >
> >> >> > Should I reset all of these to 1 ?
> >> >> > What is the purpose of this field default_nic = 0.
> >> >> >
> >> >> > vlan://500 in my case limits results only to the network for this
> VR
> >> that
> >> >> > is having problems...
> >> >> >
> >> >> > Any suggestions ?
> >> >> >
> >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> >> "gateway"
> >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> >> "isolation_uri"
> >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> >> "ip6_gateway"
> >> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> >> "06:70:0a:00:00:ac"
> >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-04
> >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05"
> "2015-02-04
> >> >> > 22:06:23" \N \N "0" "1"
> >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> >> "06:fc:c2:00:00:ad"
> >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-03
> >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35"
> "2015-03-03
> >> >> > 14:45:47" \N \N "0" "1"
> >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> >> "06:d7:ac:00:00:ae"
> >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-02-19
> >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02"
> "2015-02-19
> >> >> > 02:10:45" \N \N "0" "1"
> >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511"
> "06:b5:72:00:00:72"
> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0"
> "2014-01-27
> >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08"
> "2014-01-27
> >> >> > 13:38:52" \N \N "0" "1"
> >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> >> "06:1d:90:00:00:72"
> >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06"
> "2014-07-23
> >> >> > 08:15:18" \N \N "0" "1"
> >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> >> "06:33:ac:00:00:77"
> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25"
> "2015-03-16
> >> >> > 11:41:11" \N \N "0" "1"
> >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> >> "06:8a:ac:00:00:77"
> >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31"
> "2015-03-16
> >> >> > 12:02:59" \N \N "0" "1"
> >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> >> "06:c2:c4:00:00:7a"
> >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43"
> "2014-07-23
> >> >> > 08:16:12" \N \N "0" "1"
> >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> >> "06:1c:78:00:00:7c"
> >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44"
> "2014-07-23
> >> >> > 08:16:32" \N \N "0" "1"
> >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252"
> "06:96:4e:00:00:8e"
> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-01-13
> >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11"
> "2014-01-13
> >> >> > 22:25:44" \N \N "0" "1"
> >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> >> "06:be:24:00:00:8e"
> >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> >> 22:54:29"
> >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> >> "06:cd:d6:00:00:97"
> >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-09-02
> >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11"
> "2014-09-02
> >> >> > 09:06:56" \N \N "0" "1"
> >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> >> "06:6e:0c:00:00:9c"
> >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> >> >> 14:32:00"
> >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> >> "06:b8:b4:00:00:a1"
> >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-12-02
> >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08"
> "2014-12-02
> >> >> > 13:58:11" \N \N "0" "1"
> >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> >> "06:f8:36:00:00:a7"
> >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-01-21
> >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13"
> "2015-01-21
> >> >> > 18:46:10" \N \N "0" "1"
> >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> >> "06:ac:4e:00:00:54"
> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24"
> "2014-07-23
> >> >> > 10:33:59" \N \N "0" "1"
> >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> >> "06:46:d8:00:00:54"
> >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2014-07-23
> >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03"
> "2014-07-23
> >> >> > 14:54:11" \N \N "0" "1"
> >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102"
> "06:7c:54:00:00:35"
> >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> >> >> 12:25:41"
> >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> >> "06:c9:e0:00:00:3a"
> >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2015-03-16
> >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38"
> "2015-03-16
> >> >> > 12:03:42" \N \N "0" "1"
> >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14"
> "06:b1:30:00:00:2a"
> >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "206"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2013-01-11
> >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41"
> "2013-01-11
> >> >> > 01:11:18" \N \N "0" "1"
> >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12"
> "06:12:88:00:00:2b"
> >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "206"
> >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1"
> "2013-01-11
> >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40"
> "2013-01-11
> >> >> > 00:35:20" \N \N "0" "1"
> >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> >> "06:02:a8:00:00:e2"
> >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> >> "212"
> >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> >> >> 14:16:23"
> >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >> >
> >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for
> the
> >> >> >> router code, the null pointer indicates that there's no default
> nic
> >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> >> >> delete/move just that host you may be able to start the router,
> or at
> >> >> >> least get past this. You can also look into the db and see if you
> can
> >> >> >> find its nics in the cloud.nics table and see if there is in fact
> one
> >> >> >> marked default.
> >> >> >>
> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com>
> wrote:
> >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
> >> that.
> >> >> >> > It would suggest some inconsistent/unexpected data when
> creating a
> >> >> >> > DHCP entry for one of the guests serviced by this router. It's
> too
> >> bad
> >> >> >> > that one bad entry is fatal for the whole router.
> >> >> >> >
> >> >> >> >
> >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> >> andrija.panic@gmail.com>
> >> >> >> wrote:
> >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
> >> network
> >> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >> >>
> >> >> >> >> MGMT server sends the STOP command to agent, even though the VM
> >> was
> >> >> >> never
> >> >> >> >> started, BUT the storage provisioning from template to volume
> is
> >> >> done...
> >> >> >> >>
> >> >> >> >> We are also looking into some external help as we speak...
> >> >> >> >>
> >> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >> >>
> >> >> >> >>> Hi,
> >> >> >> >>>
> >> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> >>>
> >> >> >> >>> Nux!
> >> >> >> >>> www.nux.ro
> >> >> >> >>>
> >> >> >> >>> ----- Original Message -----
> >> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >> >>>
> >> >> >> >>> > Hi guys,
> >> >> >> >>> >
> >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS
> updated
> >> >> right
> >> >> >> >>> before
> >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >> >>> >
> >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
> >> GUEST
> >> >> >> network
> >> >> >> >>> > anymore.
> >> >> >> >>> > And some VMs are down - and cant be started because they
> >> depend on
> >> >> >> this
> >> >> >> >>> > VR...
> >> >> >> >>> >
> >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created
> fine,
> >> but
> >> >> >> this
> >> >> >> >>> one
> >> >> >> >>> > fro Guest network fails to start:
> >> >> >> >>> >
> >> >> >> >>> > Here you can see, after agent copies template from secondary
> >> >> >> storage, to
> >> >> >> >>> > primary local storage, it created base image, and backing
> file
> >> -
> >> >> so
> >> >> >> >>> storage
> >> >> >> >>> > setup seems completed.
> >> >> >> >>> >
> >> >> >> >>> > Than all out of sudden we have errors:
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >>
> >> >> >> >> Andrija Panić
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
they're just triggered by you trying to fix the situation by
migrating, but the original issue was something else entirely.
On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com> wrote:
> I did migrate and also changed accounts, unsucessfully, so some bugs
> definitively or my specific setup...
>
> Thanks, I' fixing this now and will let you know.
>
> On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>
>> Yes, each VM should have at least one default nic, so if there's only
>> one nic it should be set to default. Take a db backup first, of
>> course, before messing with it. Any idea how it may have happened? Do
>> you migrate VMs between networks ever?
>>
>> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
>> > all of them to default_nic=1... ?
>> >
>> >
>> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> >
>> >> VMs can have multiple nics and be on multiple networks. If you set a
>> >> nic as default, it becomes the network that the vm has its default
>> >> route on. Every VM should have a default nic, and if it doesn't I
>> >> wonder how it might have happened (maybe a specific combination of
>> >> add/delete nic triggered a bug?). You should set a default nic for
>> >> every VM that might be missing one, and see if that gets your router
>> >> up.
>> >>
>> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <andrija.panic@gmail.com
>> >
>> >> wrote:
>> >> > Hi Marcus,
>> >> >
>> >> > Thanks a lot fot hint
>> >> >
>> >> > True, I have the 0 as the value for some reason in database, for
>> couple
>> >> of
>> >> > NICs....
>> >> > select * from nics where ip4_address like "46.232%" and broadcast_uri
>> =
>> >> > "vlan://500" and default_nic = 0;
>> >> >
>> >> > results: http://pastebin.com/rDAe2RY9
>> >> >
>> >> > or down there...
>> >> >
>> >> > This Techvee-FileServer server is already running (still not dead)
>> and I
>> >> > can see 1 NIC from UI...
>> >> >
>> >> > Should I reset all of these to 1 ?
>> >> > What is the purpose of this field default_nic = 0.
>> >> >
>> >> > vlan://500 in my case limits results only to the network for this VR
>> that
>> >> > is having problems...
>> >> >
>> >> > Any suggestions ?
>> >> >
>> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>> "gateway"
>> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> >> "isolation_uri"
>> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> "ip6_gateway"
>> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> "06:70:0a:00:00:ac"
>> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
>> >> > 22:06:23" \N \N "0" "1"
>> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> "06:fc:c2:00:00:ad"
>> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
>> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
>> >> > 14:45:47" \N \N "0" "1"
>> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> "06:d7:ac:00:00:ae"
>> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
>> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
>> >> > 02:10:45" \N \N "0" "1"
>> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
>> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
>> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
>> >> > 13:38:52" \N \N "0" "1"
>> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> "06:1d:90:00:00:72"
>> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
>> >> > 08:15:18" \N \N "0" "1"
>> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> "06:33:ac:00:00:77"
>> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
>> >> > 11:41:11" \N \N "0" "1"
>> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> "06:8a:ac:00:00:77"
>> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
>> >> > 12:02:59" \N \N "0" "1"
>> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> "06:c2:c4:00:00:7a"
>> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
>> >> > 08:16:12" \N \N "0" "1"
>> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> "06:1c:78:00:00:7c"
>> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
>> >> > 08:16:32" \N \N "0" "1"
>> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
>> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
>> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
>> >> > 22:25:44" \N \N "0" "1"
>> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> "06:be:24:00:00:8e"
>> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> 22:54:29"
>> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> "06:cd:d6:00:00:97"
>> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
>> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
>> >> > 09:06:56" \N \N "0" "1"
>> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> "06:6e:0c:00:00:9c"
>> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> >> 14:32:00"
>> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> "06:b8:b4:00:00:a1"
>> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
>> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
>> >> > 13:58:11" \N \N "0" "1"
>> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> "06:f8:36:00:00:a7"
>> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
>> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
>> >> > 18:46:10" \N \N "0" "1"
>> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> "06:ac:4e:00:00:54"
>> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
>> >> > 10:33:59" \N \N "0" "1"
>> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> "06:46:d8:00:00:54"
>> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
>> >> > 14:54:11" \N \N "0" "1"
>> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
>> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> >> 12:25:41"
>> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> "06:c9:e0:00:00:3a"
>> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
>> >> > 12:03:42" \N \N "0" "1"
>> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
>> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "206"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
>> >> > 01:11:18" \N \N "0" "1"
>> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
>> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "206"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
>> >> > 00:35:20" \N \N "0" "1"
>> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> "06:02:a8:00:00:e2"
>> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> >> 14:16:23"
>> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >> >
>> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> >> >> router code, the null pointer indicates that there's no default nic
>> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
>> >> >> delete/move just that host you may be able to start the router, or at
>> >> >> least get past this. You can also look into the db and see if you can
>> >> >> find its nics in the cloud.nics table and see if there is in fact one
>> >> >> marked default.
>> >> >>
>> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>> that.
>> >> >> > It would suggest some inconsistent/unexpected data when creating a
>> >> >> > DHCP entry for one of the guests serviced by this router. It's too
>> bad
>> >> >> > that one bad entry is fatal for the whole router.
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> >> andrija.panic@gmail.com>
>> >> >> wrote:
>> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>> network
>> >> >> >> (Guest Shared netwotk) to VPCs...
>> >> >> >>
>> >> >> >> MGMT server sends the STOP command to agent, even though the VM
>> was
>> >> >> never
>> >> >> >> started, BUT the storage provisioning from template to volume is
>> >> done...
>> >> >> >>
>> >> >> >> We are also looking into some external help as we speak...
>> >> >> >>
>> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Have you managed to get to the bottom of this?
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >> >>>
>> >> >> >>> Nux!
>> >> >> >>> www.nux.ro
>> >> >> >>>
>> >> >> >>> ----- Original Message -----
>> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >> >>>
>> >> >> >>> > Hi guys,
>> >> >> >>> >
>> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
>> >> right
>> >> >> >>> before
>> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >> >>> >
>> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>> GUEST
>> >> >> network
>> >> >> >>> > anymore.
>> >> >> >>> > And some VMs are down - and cant be started because they
>> depend on
>> >> >> this
>> >> >> >>> > VR...
>> >> >> >>> >
>> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine,
>> but
>> >> >> this
>> >> >> >>> one
>> >> >> >>> > fro Guest network fails to start:
>> >> >> >>> >
>> >> >> >>> > Here you can see, after agent copies template from secondary
>> >> >> storage, to
>> >> >> >>> > primary local storage, it created base image, and backing file
>> -
>> >> so
>> >> >> >>> storage
>> >> >> >>> > setup seems completed.
>> >> >> >>> >
>> >> >> >>> > Than all out of sudden we have errors:
>> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >>
>> >> >> >> Andrija Panić
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Ok, just watch for those createdhcpentry mgmt server logs. Perhaps
they're just triggered by you trying to fix the situation by
migrating, but the original issue was something else entirely.
On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <an...@gmail.com> wrote:
> I did migrate and also changed accounts, unsucessfully, so some bugs
> definitively or my specific setup...
>
> Thanks, I' fixing this now and will let you know.
>
> On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
>
>> Yes, each VM should have at least one default nic, so if there's only
>> one nic it should be set to default. Take a db backup first, of
>> course, before messing with it. Any idea how it may have happened? Do
>> you migrate VMs between networks ever?
>>
>> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
>> > all of them to default_nic=1... ?
>> >
>> >
>> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>> >
>> >> VMs can have multiple nics and be on multiple networks. If you set a
>> >> nic as default, it becomes the network that the vm has its default
>> >> route on. Every VM should have a default nic, and if it doesn't I
>> >> wonder how it might have happened (maybe a specific combination of
>> >> add/delete nic triggered a bug?). You should set a default nic for
>> >> every VM that might be missing one, and see if that gets your router
>> >> up.
>> >>
>> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <andrija.panic@gmail.com
>> >
>> >> wrote:
>> >> > Hi Marcus,
>> >> >
>> >> > Thanks a lot fot hint
>> >> >
>> >> > True, I have the 0 as the value for some reason in database, for
>> couple
>> >> of
>> >> > NICs....
>> >> > select * from nics where ip4_address like "46.232%" and broadcast_uri
>> =
>> >> > "vlan://500" and default_nic = 0;
>> >> >
>> >> > results: http://pastebin.com/rDAe2RY9
>> >> >
>> >> > or down there...
>> >> >
>> >> > This Techvee-FileServer server is already running (still not dead)
>> and I
>> >> > can see 1 NIC from UI...
>> >> >
>> >> > Should I reset all of these to 1 ?
>> >> > What is the purpose of this field default_nic = 0.
>> >> >
>> >> > vlan://500 in my case limits results only to the network for this VR
>> that
>> >> > is having problems...
>> >> >
>> >> > Any suggestions ?
>> >> >
>> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
>> "gateway"
>> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> >> > "reserver_name" "reservation_id" "device_id" "update_time"
>> >> "isolation_uri"
>> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
>> "ip6_gateway"
>> >> > "ip6_cidr" "secondary_ip" "display_nic"
>> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
>> "06:70:0a:00:00:ac"
>> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
>> >> > 22:06:23" \N \N "0" "1"
>> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
>> "06:fc:c2:00:00:ad"
>> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
>> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
>> >> > 14:45:47" \N \N "0" "1"
>> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
>> "06:d7:ac:00:00:ae"
>> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
>> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
>> >> > 02:10:45" \N \N "0" "1"
>> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
>> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
>> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
>> >> > 13:38:52" \N \N "0" "1"
>> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
>> "06:1d:90:00:00:72"
>> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
>> >> > 08:15:18" \N \N "0" "1"
>> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
>> "06:33:ac:00:00:77"
>> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
>> >> > 11:41:11" \N \N "0" "1"
>> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
>> "06:8a:ac:00:00:77"
>> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
>> >> > 12:02:59" \N \N "0" "1"
>> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
>> "06:c2:c4:00:00:7a"
>> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
>> >> > 08:16:12" \N \N "0" "1"
>> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
>> "06:1c:78:00:00:7c"
>> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
>> >> > 08:16:32" \N \N "0" "1"
>> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
>> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
>> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
>> >> > 22:25:44" \N \N "0" "1"
>> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
>> "06:be:24:00:00:8e"
>> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> >> 22:54:29"
>> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
>> "06:cd:d6:00:00:97"
>> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
>> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
>> >> > 09:06:56" \N \N "0" "1"
>> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
>> "06:6e:0c:00:00:9c"
>> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> >> 14:32:00"
>> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
>> "06:b8:b4:00:00:a1"
>> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
>> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
>> >> > 13:58:11" \N \N "0" "1"
>> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
>> "06:f8:36:00:00:a7"
>> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
>> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
>> >> > 18:46:10" \N \N "0" "1"
>> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
>> "06:ac:4e:00:00:54"
>> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
>> >> > 10:33:59" \N \N "0" "1"
>> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
>> "06:46:d8:00:00:54"
>> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
>> >> > 14:54:11" \N \N "0" "1"
>> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
>> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> >> 12:25:41"
>> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
>> "06:c9:e0:00:00:3a"
>> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
>> >> > 12:03:42" \N \N "0" "1"
>> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
>> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "206"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
>> >> > 01:11:18" \N \N "0" "1"
>> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
>> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "206"
>> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
>> >> > 00:35:20" \N \N "0" "1"
>> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
>> "06:02:a8:00:00:e2"
>> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
>> "212"
>> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> >> 14:16:23"
>> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >> >
>> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> >> >> router code, the null pointer indicates that there's no default nic
>> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
>> >> >> delete/move just that host you may be able to start the router, or at
>> >> >> least get past this. You can also look into the db and see if you can
>> >> >> find its nics in the cloud.nics table and see if there is in fact one
>> >> >> marked default.
>> >> >>
>> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
>> that.
>> >> >> > It would suggest some inconsistent/unexpected data when creating a
>> >> >> > DHCP entry for one of the guests serviced by this router. It's too
>> bad
>> >> >> > that one bad entry is fatal for the whole router.
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> >> andrija.panic@gmail.com>
>> >> >> wrote:
>> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
>> network
>> >> >> >> (Guest Shared netwotk) to VPCs...
>> >> >> >>
>> >> >> >> MGMT server sends the STOP command to agent, even though the VM
>> was
>> >> >> never
>> >> >> >> started, BUT the storage provisioning from template to volume is
>> >> done...
>> >> >> >>
>> >> >> >> We are also looking into some external help as we speak...
>> >> >> >>
>> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Have you managed to get to the bottom of this?
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >> >>>
>> >> >> >>> Nux!
>> >> >> >>> www.nux.ro
>> >> >> >>>
>> >> >> >>> ----- Original Message -----
>> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >> >>>
>> >> >> >>> > Hi guys,
>> >> >> >>> >
>> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
>> >> right
>> >> >> >>> before
>> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >> >>> >
>> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
>> GUEST
>> >> >> network
>> >> >> >>> > anymore.
>> >> >> >>> > And some VMs are down - and cant be started because they
>> depend on
>> >> >> this
>> >> >> >>> > VR...
>> >> >> >>> >
>> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine,
>> but
>> >> >> this
>> >> >> >>> one
>> >> >> >>> > fro Guest network fails to start:
>> >> >> >>> >
>> >> >> >>> > Here you can see, after agent copies template from secondary
>> >> >> storage, to
>> >> >> >>> > primary local storage, it created base image, and backing file
>> -
>> >> so
>> >> >> >>> storage
>> >> >> >>> > setup seems completed.
>> >> >> >>> >
>> >> >> >>> > Than all out of sudden we have errors:
>> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >>
>> >> >> >> Andrija Panić
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I did migrate and also changed accounts, unsucessfully, so some bugs
definitively or my specific setup...
Thanks, I' fixing this now and will let you know.
On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> Yes, each VM should have at least one default nic, so if there's only
> one nic it should be set to default. Take a db backup first, of
> course, before messing with it. Any idea how it may have happened? Do
> you migrate VMs between networks ever?
>
> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
> > all of them to default_nic=1... ?
> >
> >
> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >
> >> VMs can have multiple nics and be on multiple networks. If you set a
> >> nic as default, it becomes the network that the vm has its default
> >> route on. Every VM should have a default nic, and if it doesn't I
> >> wonder how it might have happened (maybe a specific combination of
> >> add/delete nic triggered a bug?). You should set a default nic for
> >> every VM that might be missing one, and see if that gets your router
> >> up.
> >>
> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > Hi Marcus,
> >> >
> >> > Thanks a lot fot hint
> >> >
> >> > True, I have the 0 as the value for some reason in database, for
> couple
> >> of
> >> > NICs....
> >> > select * from nics where ip4_address like "46.232%" and broadcast_uri
> =
> >> > "vlan://500" and default_nic = 0;
> >> >
> >> > results: http://pastebin.com/rDAe2RY9
> >> >
> >> > or down there...
> >> >
> >> > This Techvee-FileServer server is already running (still not dead)
> and I
> >> > can see 1 NIC from UI...
> >> >
> >> > Should I reset all of these to 1 ?
> >> > What is the purpose of this field default_nic = 0.
> >> >
> >> > vlan://500 in my case limits results only to the network for this VR
> that
> >> > is having problems...
> >> >
> >> > Any suggestions ?
> >> >
> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> "gateway"
> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> "isolation_uri"
> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> "ip6_gateway"
> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> "06:70:0a:00:00:ac"
> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> >> > 22:06:23" \N \N "0" "1"
> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> "06:fc:c2:00:00:ad"
> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> >> > 14:45:47" \N \N "0" "1"
> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> "06:d7:ac:00:00:ae"
> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> >> > 02:10:45" \N \N "0" "1"
> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> >> > 13:38:52" \N \N "0" "1"
> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> "06:1d:90:00:00:72"
> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> >> > 08:15:18" \N \N "0" "1"
> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> "06:33:ac:00:00:77"
> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> >> > 11:41:11" \N \N "0" "1"
> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> "06:8a:ac:00:00:77"
> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> >> > 12:02:59" \N \N "0" "1"
> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> "06:c2:c4:00:00:7a"
> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> >> > 08:16:12" \N \N "0" "1"
> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> "06:1c:78:00:00:7c"
> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> >> > 08:16:32" \N \N "0" "1"
> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> >> > 22:25:44" \N \N "0" "1"
> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> "06:be:24:00:00:8e"
> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> 22:54:29"
> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> "06:cd:d6:00:00:97"
> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> >> > 09:06:56" \N \N "0" "1"
> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> "06:6e:0c:00:00:9c"
> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> >> 14:32:00"
> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> "06:b8:b4:00:00:a1"
> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> >> > 13:58:11" \N \N "0" "1"
> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> "06:f8:36:00:00:a7"
> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> >> > 18:46:10" \N \N "0" "1"
> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> "06:ac:4e:00:00:54"
> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> >> > 10:33:59" \N \N "0" "1"
> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> "06:46:d8:00:00:54"
> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> >> > 14:54:11" \N \N "0" "1"
> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> >> 12:25:41"
> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> "06:c9:e0:00:00:3a"
> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> >> > 12:03:42" \N \N "0" "1"
> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "206"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> >> > 01:11:18" \N \N "0" "1"
> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "206"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> >> > 00:35:20" \N \N "0" "1"
> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> "06:02:a8:00:00:e2"
> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> >> 14:16:23"
> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >> >
> >> >
> >> >
> >> >
> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> >> >> router code, the null pointer indicates that there's no default nic
> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> >> delete/move just that host you may be able to start the router, or at
> >> >> least get past this. You can also look into the db and see if you can
> >> >> find its nics in the cloud.nics table and see if there is in fact one
> >> >> marked default.
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
> that.
> >> >> > It would suggest some inconsistent/unexpected data when creating a
> >> >> > DHCP entry for one of the guests serviced by this router. It's too
> bad
> >> >> > that one bad entry is fatal for the whole router.
> >> >> >
> >> >> >
> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
> network
> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >>
> >> >> >> MGMT server sends the STOP command to agent, even though the VM
> was
> >> >> never
> >> >> >> started, BUT the storage provisioning from template to volume is
> >> done...
> >> >> >>
> >> >> >> We are also looking into some external help as we speak...
> >> >> >>
> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >>>
> >> >> >>> --
> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >>>
> >> >> >>> Nux!
> >> >> >>> www.nux.ro
> >> >> >>>
> >> >> >>> ----- Original Message -----
> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >>>
> >> >> >>> > Hi guys,
> >> >> >>> >
> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> >> right
> >> >> >>> before
> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >>> >
> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
> GUEST
> >> >> network
> >> >> >>> > anymore.
> >> >> >>> > And some VMs are down - and cant be started because they
> depend on
> >> >> this
> >> >> >>> > VR...
> >> >> >>> >
> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine,
> but
> >> >> this
> >> >> >>> one
> >> >> >>> > fro Guest network fails to start:
> >> >> >>> >
> >> >> >>> > Here you can see, after agent copies template from secondary
> >> >> storage, to
> >> >> >>> > primary local storage, it created base image, and backing file
> -
> >> so
> >> >> >>> storage
> >> >> >>> > setup seems completed.
> >> >> >>> >
> >> >> >>> > Than all out of sudden we have errors:
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >>
> >> >> >> Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I did migrate and also changed accounts, unsucessfully, so some bugs
definitively or my specific setup...
Thanks, I' fixing this now and will let you know.
On 16 March 2015 at 16:42, Marcus <sh...@gmail.com> wrote:
> Yes, each VM should have at least one default nic, so if there's only
> one nic it should be set to default. Take a db backup first, of
> course, before messing with it. Any idea how it may have happened? Do
> you migrate VMs between networks ever?
>
> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
> > all of them to default_nic=1... ?
> >
> >
> > On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> >
> >> VMs can have multiple nics and be on multiple networks. If you set a
> >> nic as default, it becomes the network that the vm has its default
> >> route on. Every VM should have a default nic, and if it doesn't I
> >> wonder how it might have happened (maybe a specific combination of
> >> add/delete nic triggered a bug?). You should set a default nic for
> >> every VM that might be missing one, and see if that gets your router
> >> up.
> >>
> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > Hi Marcus,
> >> >
> >> > Thanks a lot fot hint
> >> >
> >> > True, I have the 0 as the value for some reason in database, for
> couple
> >> of
> >> > NICs....
> >> > select * from nics where ip4_address like "46.232%" and broadcast_uri
> =
> >> > "vlan://500" and default_nic = 0;
> >> >
> >> > results: http://pastebin.com/rDAe2RY9
> >> >
> >> > or down there...
> >> >
> >> > This Techvee-FileServer server is already running (still not dead)
> and I
> >> > can see 1 NIC from UI...
> >> >
> >> > Should I reset all of these to 1 ?
> >> > What is the purpose of this field default_nic = 0.
> >> >
> >> > vlan://500 in my case limits results only to the network for this VR
> that
> >> > is having problems...
> >> >
> >> > Any suggestions ?
> >> >
> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask"
> "gateway"
> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> >> > "reserver_name" "reservation_id" "device_id" "update_time"
> >> "isolation_uri"
> >> > "ip6_address" "default_nic" "vm_type" "created" "removed"
> "ip6_gateway"
> >> > "ip6_cidr" "secondary_ip" "display_nic"
> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795"
> "06:70:0a:00:00:ac"
> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> >> > 22:06:23" \N \N "0" "1"
> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958"
> "06:fc:c2:00:00:ad"
> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> >> > 14:45:47" \N \N "0" "1"
> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962"
> "06:d7:ac:00:00:ae"
> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> >> > 02:10:45" \N \N "0" "1"
> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> >> > 13:38:52" \N \N "0" "1"
> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088"
> "06:1d:90:00:00:72"
> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> >> > 08:15:18" \N \N "0" "1"
> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306"
> "06:33:ac:00:00:77"
> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> >> > 11:41:11" \N \N "0" "1"
> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311"
> "06:8a:ac:00:00:77"
> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> >> > 12:02:59" \N \N "0" "1"
> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089"
> "06:c2:c4:00:00:7a"
> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> >> > 08:16:12" \N \N "0" "1"
> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090"
> "06:1c:78:00:00:7c"
> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> >> > 08:16:32" \N \N "0" "1"
> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> >> > 22:25:44" \N \N "0" "1"
> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797"
> "06:be:24:00:00:8e"
> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> >> 22:54:29"
> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213"
> "06:cd:d6:00:00:97"
> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> >> > 09:06:56" \N \N "0" "1"
> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304"
> "06:6e:0c:00:00:9c"
> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> >> 14:32:00"
> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488"
> "06:b8:b4:00:00:a1"
> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> >> > 13:58:11" \N \N "0" "1"
> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706"
> "06:f8:36:00:00:a7"
> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> >> > 18:46:10" \N \N "0" "1"
> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102"
> "06:ac:4e:00:00:54"
> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> >> > 10:33:59" \N \N "0" "1"
> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108"
> "06:46:d8:00:00:54"
> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> >> > 14:54:11" \N \N "0" "1"
> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> >> 12:25:41"
> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312"
> "06:c9:e0:00:00:3a"
> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> >> > 12:03:42" \N \N "0" "1"
> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "206"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> >> > 01:11:18" \N \N "0" "1"
> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "206"
> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> >> > 00:35:20" \N \N "0" "1"
> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871"
> "06:02:a8:00:00:e2"
> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500"
> "212"
> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> >> 14:16:23"
> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >> >
> >> >
> >> >
> >> >
> >> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >> >
> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> >> >> router code, the null pointer indicates that there's no default nic
> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> >> delete/move just that host you may be able to start the router, or at
> >> >> least get past this. You can also look into the db and see if you can
> >> >> find its nics in the cloud.nics table and see if there is in fact one
> >> >> marked default.
> >> >>
> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after
> that.
> >> >> > It would suggest some inconsistent/unexpected data when creating a
> >> >> > DHCP entry for one of the guests serviced by this router. It's too
> bad
> >> >> > that one bad entry is fatal for the whole router.
> >> >> >
> >> >> >
> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> >> andrija.panic@gmail.com>
> >> >> wrote:
> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS
> network
> >> >> >> (Guest Shared netwotk) to VPCs...
> >> >> >>
> >> >> >> MGMT server sends the STOP command to agent, even though the VM
> was
> >> >> never
> >> >> >> started, BUT the storage provisioning from template to volume is
> >> done...
> >> >> >>
> >> >> >> We are also looking into some external help as we speak...
> >> >> >>
> >> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Have you managed to get to the bottom of this?
> >> >> >>>
> >> >> >>> --
> >> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >> >>>
> >> >> >>> Nux!
> >> >> >>> www.nux.ro
> >> >> >>>
> >> >> >>> ----- Original Message -----
> >> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >> >>>
> >> >> >>> > Hi guys,
> >> >> >>> >
> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> >> right
> >> >> >>> before
> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >> >>> >
> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED
> GUEST
> >> >> network
> >> >> >>> > anymore.
> >> >> >>> > And some VMs are down - and cant be started because they
> depend on
> >> >> this
> >> >> >>> > VR...
> >> >> >>> >
> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine,
> but
> >> >> this
> >> >> >>> one
> >> >> >>> > fro Guest network fails to start:
> >> >> >>> >
> >> >> >>> > Here you can see, after agent copies template from secondary
> >> >> storage, to
> >> >> >>> > primary local storage, it created base image, and backing file
> -
> >> so
> >> >> >>> storage
> >> >> >>> > setup seems completed.
> >> >> >>> >
> >> >> >>> > Than all out of sudden we have errors:
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >>
> >> >> >> Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Yes, each VM should have at least one default nic, so if there's only
one nic it should be set to default. Take a db backup first, of
course, before messing with it. Any idea how it may have happened? Do
you migrate VMs between networks ever?
On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com> wrote:
> Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
> all of them to default_nic=1... ?
>
>
> On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>
>> VMs can have multiple nics and be on multiple networks. If you set a
>> nic as default, it becomes the network that the vm has its default
>> route on. Every VM should have a default nic, and if it doesn't I
>> wonder how it might have happened (maybe a specific combination of
>> add/delete nic triggered a bug?). You should set a default nic for
>> every VM that might be missing one, and see if that gets your router
>> up.
>>
>> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Hi Marcus,
>> >
>> > Thanks a lot fot hint
>> >
>> > True, I have the 0 as the value for some reason in database, for couple
>> of
>> > NICs....
>> > select * from nics where ip4_address like "46.232%" and broadcast_uri =
>> > "vlan://500" and default_nic = 0;
>> >
>> > results: http://pastebin.com/rDAe2RY9
>> >
>> > or down there...
>> >
>> > This Techvee-FileServer server is already running (still not dead) and I
>> > can see 1 NIC from UI...
>> >
>> > Should I reset all of these to 1 ?
>> > What is the purpose of this field default_nic = 0.
>> >
>> > vlan://500 in my case limits results only to the network for this VR that
>> > is having problems...
>> >
>> > Any suggestions ?
>> >
>> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
>> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> > "reserver_name" "reservation_id" "device_id" "update_time"
>> "isolation_uri"
>> > "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
>> > "ip6_cidr" "secondary_ip" "display_nic"
>> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
>> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
>> > 22:06:23" \N \N "0" "1"
>> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
>> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
>> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
>> > 14:45:47" \N \N "0" "1"
>> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
>> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
>> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
>> > 02:10:45" \N \N "0" "1"
>> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
>> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
>> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
>> > 13:38:52" \N \N "0" "1"
>> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
>> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
>> > 08:15:18" \N \N "0" "1"
>> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
>> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
>> > 11:41:11" \N \N "0" "1"
>> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
>> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
>> > 12:02:59" \N \N "0" "1"
>> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
>> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
>> > 08:16:12" \N \N "0" "1"
>> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
>> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
>> > 08:16:32" \N \N "0" "1"
>> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
>> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
>> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
>> > 22:25:44" \N \N "0" "1"
>> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
>> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> 22:54:29"
>> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
>> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
>> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
>> > 09:06:56" \N \N "0" "1"
>> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
>> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> 14:32:00"
>> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
>> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
>> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
>> > 13:58:11" \N \N "0" "1"
>> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
>> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
>> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
>> > 18:46:10" \N \N "0" "1"
>> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
>> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
>> > 10:33:59" \N \N "0" "1"
>> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
>> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
>> > 14:54:11" \N \N "0" "1"
>> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
>> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> 12:25:41"
>> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
>> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
>> > 12:03:42" \N \N "0" "1"
>> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
>> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
>> > 01:11:18" \N \N "0" "1"
>> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
>> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
>> > 00:35:20" \N \N "0" "1"
>> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
>> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> 14:16:23"
>> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >
>> >
>> >
>> >
>> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >
>> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> >> router code, the null pointer indicates that there's no default nic
>> >> for one of your guests "Techvee-FileServer". I'd guess that if you
>> >> delete/move just that host you may be able to start the router, or at
>> >> least get past this. You can also look into the db and see if you can
>> >> find its nics in the cloud.nics table and see if there is in fact one
>> >> marked default.
>> >>
>> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> >> > either "Techvee-FileServer" or the DHCP entry immediately after that.
>> >> > It would suggest some inconsistent/unexpected data when creating a
>> >> > DHCP entry for one of the guests serviced by this router. It's too bad
>> >> > that one bad entry is fatal for the whole router.
>> >> >
>> >> >
>> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> andrija.panic@gmail.com>
>> >> wrote:
>> >> >> Not really - we are painfully migrating stopped VMs, from VPS network
>> >> >> (Guest Shared netwotk) to VPCs...
>> >> >>
>> >> >> MGMT server sends the STOP command to agent, even though the VM was
>> >> never
>> >> >> started, BUT the storage provisioning from template to volume is
>> done...
>> >> >>
>> >> >> We are also looking into some external help as we speak...
>> >> >>
>> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Have you managed to get to the bottom of this?
>> >> >>>
>> >> >>> --
>> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >>>
>> >> >>> Nux!
>> >> >>> www.nux.ro
>> >> >>>
>> >> >>> ----- Original Message -----
>> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >>>
>> >> >>> > Hi guys,
>> >> >>> >
>> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
>> right
>> >> >>> before
>> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >>> >
>> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
>> >> network
>> >> >>> > anymore.
>> >> >>> > And some VMs are down - and cant be started because they depend on
>> >> this
>> >> >>> > VR...
>> >> >>> >
>> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
>> >> this
>> >> >>> one
>> >> >>> > fro Guest network fails to start:
>> >> >>> >
>> >> >>> > Here you can see, after agent copies template from secondary
>> >> storage, to
>> >> >>> > primary local storage, it created base image, and backing file -
>> so
>> >> >>> storage
>> >> >>> > setup seems completed.
>> >> >>> >
>> >> >>> > Than all out of sudden we have errors:
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Yes, each VM should have at least one default nic, so if there's only
one nic it should be set to default. Take a db backup first, of
course, before messing with it. Any idea how it may have happened? Do
you migrate VMs between networks ever?
On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <an...@gmail.com> wrote:
> Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
> all of them to default_nic=1... ?
>
>
> On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
>
>> VMs can have multiple nics and be on multiple networks. If you set a
>> nic as default, it becomes the network that the vm has its default
>> route on. Every VM should have a default nic, and if it doesn't I
>> wonder how it might have happened (maybe a specific combination of
>> add/delete nic triggered a bug?). You should set a default nic for
>> every VM that might be missing one, and see if that gets your router
>> up.
>>
>> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Hi Marcus,
>> >
>> > Thanks a lot fot hint
>> >
>> > True, I have the 0 as the value for some reason in database, for couple
>> of
>> > NICs....
>> > select * from nics where ip4_address like "46.232%" and broadcast_uri =
>> > "vlan://500" and default_nic = 0;
>> >
>> > results: http://pastebin.com/rDAe2RY9
>> >
>> > or down there...
>> >
>> > This Techvee-FileServer server is already running (still not dead) and I
>> > can see 1 NIC from UI...
>> >
>> > Should I reset all of these to 1 ?
>> > What is the purpose of this field default_nic = 0.
>> >
>> > vlan://500 in my case limits results only to the network for this VR that
>> > is having problems...
>> >
>> > Any suggestions ?
>> >
>> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
>> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
>> > "reserver_name" "reservation_id" "device_id" "update_time"
>> "isolation_uri"
>> > "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
>> > "ip6_cidr" "secondary_ip" "display_nic"
>> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
>> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
>> > 22:06:23" \N \N "0" "1"
>> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
>> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
>> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
>> > 14:45:47" \N \N "0" "1"
>> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
>> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
>> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
>> > 02:10:45" \N \N "0" "1"
>> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
>> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
>> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
>> > 13:38:52" \N \N "0" "1"
>> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
>> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
>> > 08:15:18" \N \N "0" "1"
>> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
>> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
>> > 11:41:11" \N \N "0" "1"
>> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
>> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
>> > 12:02:59" \N \N "0" "1"
>> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
>> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
>> > 08:16:12" \N \N "0" "1"
>> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
>> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
>> > 08:16:32" \N \N "0" "1"
>> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
>> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
>> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
>> > 22:25:44" \N \N "0" "1"
>> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
>> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
>> 22:54:29"
>> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
>> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
>> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
>> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
>> > 09:06:56" \N \N "0" "1"
>> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
>> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
>> 14:32:00"
>> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
>> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
>> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
>> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
>> > 13:58:11" \N \N "0" "1"
>> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
>> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
>> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
>> > 18:46:10" \N \N "0" "1"
>> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
>> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
>> > 10:33:59" \N \N "0" "1"
>> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
>> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
>> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
>> > 14:54:11" \N \N "0" "1"
>> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
>> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
>> 12:25:41"
>> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
>> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
>> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
>> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
>> > 12:03:42" \N \N "0" "1"
>> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
>> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
>> > 01:11:18" \N \N "0" "1"
>> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
>> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
>> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
>> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
>> > 00:35:20" \N \N "0" "1"
>> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
>> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
>> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
>> 14:16:23"
>> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>> >
>> >
>> >
>> >
>> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>> >
>> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> >> router code, the null pointer indicates that there's no default nic
>> >> for one of your guests "Techvee-FileServer". I'd guess that if you
>> >> delete/move just that host you may be able to start the router, or at
>> >> least get past this. You can also look into the db and see if you can
>> >> find its nics in the cloud.nics table and see if there is in fact one
>> >> marked default.
>> >>
>> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> >> > either "Techvee-FileServer" or the DHCP entry immediately after that.
>> >> > It would suggest some inconsistent/unexpected data when creating a
>> >> > DHCP entry for one of the guests serviced by this router. It's too bad
>> >> > that one bad entry is fatal for the whole router.
>> >> >
>> >> >
>> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
>> andrija.panic@gmail.com>
>> >> wrote:
>> >> >> Not really - we are painfully migrating stopped VMs, from VPS network
>> >> >> (Guest Shared netwotk) to VPCs...
>> >> >>
>> >> >> MGMT server sends the STOP command to agent, even though the VM was
>> >> never
>> >> >> started, BUT the storage provisioning from template to volume is
>> done...
>> >> >>
>> >> >> We are also looking into some external help as we speak...
>> >> >>
>> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Have you managed to get to the bottom of this?
>> >> >>>
>> >> >>> --
>> >> >>> Sent from the Delta quadrant using Borg technology!
>> >> >>>
>> >> >>> Nux!
>> >> >>> www.nux.ro
>> >> >>>
>> >> >>> ----- Original Message -----
>> >> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >> >>> > Subject: [URGENT - HELP NEEDED]
>> >> >>>
>> >> >>> > Hi guys,
>> >> >>> >
>> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
>> right
>> >> >>> before
>> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >> >>> >
>> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
>> >> network
>> >> >>> > anymore.
>> >> >>> > And some VMs are down - and cant be started because they depend on
>> >> this
>> >> >>> > VR...
>> >> >>> >
>> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
>> >> this
>> >> >>> one
>> >> >>> > fro Guest network fails to start:
>> >> >>> >
>> >> >>> > Here you can see, after agent copies template from secondary
>> >> storage, to
>> >> >>> > primary local storage, it created base image, and backing file -
>> so
>> >> >>> storage
>> >> >>> > setup seems completed.
>> >> >>> >
>> >> >>> > Than all out of sudden we have errors:
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
all of them to default_nic=1... ?
On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> VMs can have multiple nics and be on multiple networks. If you set a
> nic as default, it becomes the network that the vm has its default
> route on. Every VM should have a default nic, and if it doesn't I
> wonder how it might have happened (maybe a specific combination of
> add/delete nic triggered a bug?). You should set a default nic for
> every VM that might be missing one, and see if that gets your router
> up.
>
> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Hi Marcus,
> >
> > Thanks a lot fot hint
> >
> > True, I have the 0 as the value for some reason in database, for couple
> of
> > NICs....
> > select * from nics where ip4_address like "46.232%" and broadcast_uri =
> > "vlan://500" and default_nic = 0;
> >
> > results: http://pastebin.com/rDAe2RY9
> >
> > or down there...
> >
> > This Techvee-FileServer server is already running (still not dead) and I
> > can see 1 NIC from UI...
> >
> > Should I reset all of these to 1 ?
> > What is the purpose of this field default_nic = 0.
> >
> > vlan://500 in my case limits results only to the network for this VR that
> > is having problems...
> >
> > Any suggestions ?
> >
> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> > "reserver_name" "reservation_id" "device_id" "update_time"
> "isolation_uri"
> > "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
> > "ip6_cidr" "secondary_ip" "display_nic"
> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> > 22:06:23" \N \N "0" "1"
> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> > 14:45:47" \N \N "0" "1"
> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> > 02:10:45" \N \N "0" "1"
> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> > 13:38:52" \N \N "0" "1"
> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> > 08:15:18" \N \N "0" "1"
> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> > 11:41:11" \N \N "0" "1"
> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> > 12:02:59" \N \N "0" "1"
> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> > 08:16:12" \N \N "0" "1"
> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> > 08:16:32" \N \N "0" "1"
> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> > 22:25:44" \N \N "0" "1"
> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> 22:54:29"
> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> > 09:06:56" \N \N "0" "1"
> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> 14:32:00"
> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> > 13:58:11" \N \N "0" "1"
> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> > 18:46:10" \N \N "0" "1"
> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> > 10:33:59" \N \N "0" "1"
> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> > 14:54:11" \N \N "0" "1"
> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> 12:25:41"
> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> > 12:03:42" \N \N "0" "1"
> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> > 01:11:18" \N \N "0" "1"
> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> > 00:35:20" \N \N "0" "1"
> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> 14:16:23"
> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >
> >
> >
> >
> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >
> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> >> router code, the null pointer indicates that there's no default nic
> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> delete/move just that host you may be able to start the router, or at
> >> least get past this. You can also look into the db and see if you can
> >> find its nics in the cloud.nics table and see if there is in fact one
> >> marked default.
> >>
> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> > either "Techvee-FileServer" or the DHCP entry immediately after that.
> >> > It would suggest some inconsistent/unexpected data when creating a
> >> > DHCP entry for one of the guests serviced by this router. It's too bad
> >> > that one bad entry is fatal for the whole router.
> >> >
> >> >
> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> andrija.panic@gmail.com>
> >> wrote:
> >> >> Not really - we are painfully migrating stopped VMs, from VPS network
> >> >> (Guest Shared netwotk) to VPCs...
> >> >>
> >> >> MGMT server sends the STOP command to agent, even though the VM was
> >> never
> >> >> started, BUT the storage provisioning from template to volume is
> done...
> >> >>
> >> >> We are also looking into some external help as we speak...
> >> >>
> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Have you managed to get to the bottom of this?
> >> >>>
> >> >>> --
> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >>>
> >> >>> Nux!
> >> >>> www.nux.ro
> >> >>>
> >> >>> ----- Original Message -----
> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >>>
> >> >>> > Hi guys,
> >> >>> >
> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> right
> >> >>> before
> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >>> >
> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> >> network
> >> >>> > anymore.
> >> >>> > And some VMs are down - and cant be started because they depend on
> >> this
> >> >>> > VR...
> >> >>> >
> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
> >> this
> >> >>> one
> >> >>> > fro Guest network fails to start:
> >> >>> >
> >> >>> > Here you can see, after agent copies template from secondary
> >> storage, to
> >> >>> > primary local storage, it created base image, and backing file -
> so
> >> >>> storage
> >> >>> > setup seems completed.
> >> >>> >
> >> >>> > Than all out of sudden we have errors:
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change
all of them to default_nic=1... ?
On 16 March 2015 at 16:38, Marcus <sh...@gmail.com> wrote:
> VMs can have multiple nics and be on multiple networks. If you set a
> nic as default, it becomes the network that the vm has its default
> route on. Every VM should have a default nic, and if it doesn't I
> wonder how it might have happened (maybe a specific combination of
> add/delete nic triggered a bug?). You should set a default nic for
> every VM that might be missing one, and see if that gets your router
> up.
>
> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Hi Marcus,
> >
> > Thanks a lot fot hint
> >
> > True, I have the 0 as the value for some reason in database, for couple
> of
> > NICs....
> > select * from nics where ip4_address like "46.232%" and broadcast_uri =
> > "vlan://500" and default_nic = 0;
> >
> > results: http://pastebin.com/rDAe2RY9
> >
> > or down there...
> >
> > This Techvee-FileServer server is already running (still not dead) and I
> > can see 1 NIC from UI...
> >
> > Should I reset all of these to 1 ?
> > What is the purpose of this field default_nic = 0.
> >
> > vlan://500 in my case limits results only to the network for this VR that
> > is having problems...
> >
> > Any suggestions ?
> >
> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> > "reserver_name" "reservation_id" "device_id" "update_time"
> "isolation_uri"
> > "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
> > "ip6_cidr" "secondary_ip" "display_nic"
> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> > 22:06:23" \N \N "0" "1"
> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> > 14:45:47" \N \N "0" "1"
> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> > 02:10:45" \N \N "0" "1"
> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> > 13:38:52" \N \N "0" "1"
> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> > 08:15:18" \N \N "0" "1"
> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> > 11:41:11" \N \N "0" "1"
> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> > 12:02:59" \N \N "0" "1"
> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> > 08:16:12" \N \N "0" "1"
> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> > 08:16:32" \N \N "0" "1"
> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> > 22:25:44" \N \N "0" "1"
> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> 22:54:29"
> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> > 09:06:56" \N \N "0" "1"
> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11
> 14:32:00"
> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> > 13:58:11" \N \N "0" "1"
> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> > 18:46:10" \N \N "0" "1"
> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> > 10:33:59" \N \N "0" "1"
> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> > 14:54:11" \N \N "0" "1"
> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16
> 12:25:41"
> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> > 12:03:42" \N \N "0" "1"
> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> > 01:11:18" \N \N "0" "1"
> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> > 00:35:20" \N \N "0" "1"
> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12
> 14:16:23"
> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
> >
> >
> >
> >
> > On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> >
> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> >> router code, the null pointer indicates that there's no default nic
> >> for one of your guests "Techvee-FileServer". I'd guess that if you
> >> delete/move just that host you may be able to start the router, or at
> >> least get past this. You can also look into the db and see if you can
> >> find its nics in the cloud.nics table and see if there is in fact one
> >> marked default.
> >>
> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> >> > Looks like the issue is that null pointer in CreateDhcpEntry for
> >> > either "Techvee-FileServer" or the DHCP entry immediately after that.
> >> > It would suggest some inconsistent/unexpected data when creating a
> >> > DHCP entry for one of the guests serviced by this router. It's too bad
> >> > that one bad entry is fatal for the whole router.
> >> >
> >> >
> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <
> andrija.panic@gmail.com>
> >> wrote:
> >> >> Not really - we are painfully migrating stopped VMs, from VPS network
> >> >> (Guest Shared netwotk) to VPCs...
> >> >>
> >> >> MGMT server sends the STOP command to agent, even though the VM was
> >> never
> >> >> started, BUT the storage provisioning from template to volume is
> done...
> >> >>
> >> >> We are also looking into some external help as we speak...
> >> >>
> >> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Have you managed to get to the bottom of this?
> >> >>>
> >> >>> --
> >> >>> Sent from the Delta quadrant using Borg technology!
> >> >>>
> >> >>> Nux!
> >> >>> www.nux.ro
> >> >>>
> >> >>> ----- Original Message -----
> >> >>> > From: "Andrija Panic" <an...@gmail.com>
> >> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >> >>> > Subject: [URGENT - HELP NEEDED]
> >> >>>
> >> >>> > Hi guys,
> >> >>> >
> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> right
> >> >>> before
> >> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >> >>> >
> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> >> network
> >> >>> > anymore.
> >> >>> > And some VMs are down - and cant be started because they depend on
> >> this
> >> >>> > VR...
> >> >>> >
> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
> >> this
> >> >>> one
> >> >>> > fro Guest network fails to start:
> >> >>> >
> >> >>> > Here you can see, after agent copies template from secondary
> >> storage, to
> >> >>> > primary local storage, it created base image, and backing file -
> so
> >> >>> storage
> >> >>> > setup seems completed.
> >> >>> >
> >> >>> > Than all out of sudden we have errors:
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
VMs can have multiple nics and be on multiple networks. If you set a
nic as default, it becomes the network that the vm has its default
route on. Every VM should have a default nic, and if it doesn't I
wonder how it might have happened (maybe a specific combination of
add/delete nic triggered a bug?). You should set a default nic for
every VM that might be missing one, and see if that gets your router
up.
On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com> wrote:
> Hi Marcus,
>
> Thanks a lot fot hint
>
> True, I have the 0 as the value for some reason in database, for couple of
> NICs....
> select * from nics where ip4_address like "46.232%" and broadcast_uri =
> "vlan://500" and default_nic = 0;
>
> results: http://pastebin.com/rDAe2RY9
>
> or down there...
>
> This Techvee-FileServer server is already running (still not dead) and I
> can see 1 NIC from UI...
>
> Should I reset all of these to 1 ?
> What is the purpose of this field default_nic = 0.
>
> vlan://500 in my case limits results only to the network for this VR that
> is having problems...
>
> Any suggestions ?
>
> "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
> "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> "reserver_name" "reservation_id" "device_id" "update_time" "isolation_uri"
> "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
> "ip6_cidr" "secondary_ip" "display_nic"
> "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
> "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> 22:06:23" \N \N "0" "1"
> "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
> "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> 14:45:47" \N \N "0" "1"
> "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
> "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> 02:10:45" \N \N "0" "1"
> "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> 13:38:52" \N \N "0" "1"
> "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
> "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> 08:15:18" \N \N "0" "1"
> "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
> "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> 11:41:11" \N \N "0" "1"
> "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
> "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> 12:02:59" \N \N "0" "1"
> "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
> "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> 08:16:12" \N \N "0" "1"
> "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
> "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> 08:16:32" \N \N "0" "1"
> "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> 22:25:44" \N \N "0" "1"
> "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
> "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 22:54:29"
> "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
> "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> 09:06:56" \N \N "0" "1"
> "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
> "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11 14:32:00"
> "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
> "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> 13:58:11" \N \N "0" "1"
> "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
> "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> 18:46:10" \N \N "0" "1"
> "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
> "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> 10:33:59" \N \N "0" "1"
> "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
> "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> 14:54:11" \N \N "0" "1"
> "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16 12:25:41"
> "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
> "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> 12:03:42" \N \N "0" "1"
> "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> 01:11:18" \N \N "0" "1"
> "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> 00:35:20" \N \N "0" "1"
> "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
> "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12 14:16:23"
> "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>
>
>
>
> On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>
>> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> router code, the null pointer indicates that there's no default nic
>> for one of your guests "Techvee-FileServer". I'd guess that if you
>> delete/move just that host you may be able to start the router, or at
>> least get past this. You can also look into the db and see if you can
>> find its nics in the cloud.nics table and see if there is in fact one
>> marked default.
>>
>> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> > either "Techvee-FileServer" or the DHCP entry immediately after that.
>> > It would suggest some inconsistent/unexpected data when creating a
>> > DHCP entry for one of the guests serviced by this router. It's too bad
>> > that one bad entry is fatal for the whole router.
>> >
>> >
>> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> >> Not really - we are painfully migrating stopped VMs, from VPS network
>> >> (Guest Shared netwotk) to VPCs...
>> >>
>> >> MGMT server sends the STOP command to agent, even though the VM was
>> never
>> >> started, BUT the storage provisioning from template to volume is done...
>> >>
>> >> We are also looking into some external help as we speak...
>> >>
>> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Have you managed to get to the bottom of this?
>> >>>
>> >>> --
>> >>> Sent from the Delta quadrant using Borg technology!
>> >>>
>> >>> Nux!
>> >>> www.nux.ro
>> >>>
>> >>> ----- Original Message -----
>> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >>> > Subject: [URGENT - HELP NEEDED]
>> >>>
>> >>> > Hi guys,
>> >>> >
>> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>> >>> before
>> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >>> >
>> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
>> network
>> >>> > anymore.
>> >>> > And some VMs are down - and cant be started because they depend on
>> this
>> >>> > VR...
>> >>> >
>> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
>> this
>> >>> one
>> >>> > fro Guest network fails to start:
>> >>> >
>> >>> > Here you can see, after agent copies template from secondary
>> storage, to
>> >>> > primary local storage, it created base image, and backing file - so
>> >>> storage
>> >>> > setup seems completed.
>> >>> >
>> >>> > Than all out of sudden we have errors:
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
VMs can have multiple nics and be on multiple networks. If you set a
nic as default, it becomes the network that the vm has its default
route on. Every VM should have a default nic, and if it doesn't I
wonder how it might have happened (maybe a specific combination of
add/delete nic triggered a bug?). You should set a default nic for
every VM that might be missing one, and see if that gets your router
up.
On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <an...@gmail.com> wrote:
> Hi Marcus,
>
> Thanks a lot fot hint
>
> True, I have the 0 as the value for some reason in database, for couple of
> NICs....
> select * from nics where ip4_address like "46.232%" and broadcast_uri =
> "vlan://500" and default_nic = 0;
>
> results: http://pastebin.com/rDAe2RY9
>
> or down there...
>
> This Techvee-FileServer server is already running (still not dead) and I
> can see 1 NIC from UI...
>
> Should I reset all of these to 1 ?
> What is the purpose of this field default_nic = 0.
>
> vlan://500 in my case limits results only to the network for this VR that
> is having problems...
>
> Any suggestions ?
>
> "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
> "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
> "reserver_name" "reservation_id" "device_id" "update_time" "isolation_uri"
> "ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
> "ip6_cidr" "secondary_ip" "display_nic"
> "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
> "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
> 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
> 22:06:23" \N \N "0" "1"
> "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
> "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
> 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
> 14:45:47" \N \N "0" "1"
> "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
> "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
> 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
> 02:10:45" \N \N "0" "1"
> "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
> "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
> 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
> 13:38:52" \N \N "0" "1"
> "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
> "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
> 08:15:18" \N \N "0" "1"
> "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
> "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
> 11:41:11" \N \N "0" "1"
> "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
> "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
> 12:02:59" \N \N "0" "1"
> "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
> "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
> 08:16:12" \N \N "0" "1"
> "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
> "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
> 08:16:32" \N \N "0" "1"
> "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
> "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
> 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
> 22:25:44" \N \N "0" "1"
> "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
> "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 22:54:29"
> "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
> "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
> "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
> 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
> 09:06:56" \N \N "0" "1"
> "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
> "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11 14:32:00"
> "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
> "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
> "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
> 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
> 13:58:11" \N \N "0" "1"
> "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
> "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
> 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
> 18:46:10" \N \N "0" "1"
> "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
> "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
> 10:33:59" \N \N "0" "1"
> "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
> "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
> 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
> 14:54:11" \N \N "0" "1"
> "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
> "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16 12:25:41"
> "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
> "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
> "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
> 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
> 12:03:42" \N \N "0" "1"
> "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
> "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
> 01:11:18" \N \N "0" "1"
> "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
> "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
> "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
> 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
> 00:35:20" \N \N "0" "1"
> "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
> "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
> "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12 14:16:23"
> "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
>
>
>
>
> On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
>
>> FWIW, if your 4.3.2 is the same as what's in the source tree for the
>> router code, the null pointer indicates that there's no default nic
>> for one of your guests "Techvee-FileServer". I'd guess that if you
>> delete/move just that host you may be able to start the router, or at
>> least get past this. You can also look into the db and see if you can
>> find its nics in the cloud.nics table and see if there is in fact one
>> marked default.
>>
>> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
>> > Looks like the issue is that null pointer in CreateDhcpEntry for
>> > either "Techvee-FileServer" or the DHCP entry immediately after that.
>> > It would suggest some inconsistent/unexpected data when creating a
>> > DHCP entry for one of the guests serviced by this router. It's too bad
>> > that one bad entry is fatal for the whole router.
>> >
>> >
>> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> >> Not really - we are painfully migrating stopped VMs, from VPS network
>> >> (Guest Shared netwotk) to VPCs...
>> >>
>> >> MGMT server sends the STOP command to agent, even though the VM was
>> never
>> >> started, BUT the storage provisioning from template to volume is done...
>> >>
>> >> We are also looking into some external help as we speak...
>> >>
>> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Have you managed to get to the bottom of this?
>> >>>
>> >>> --
>> >>> Sent from the Delta quadrant using Borg technology!
>> >>>
>> >>> Nux!
>> >>> www.nux.ro
>> >>>
>> >>> ----- Original Message -----
>> >>> > From: "Andrija Panic" <an...@gmail.com>
>> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> >>> > Sent: Sunday, 15 March, 2015 16:19:07
>> >>> > Subject: [URGENT - HELP NEEDED]
>> >>>
>> >>> > Hi guys,
>> >>> >
>> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>> >>> before
>> >>> > that, from CentOS 6.5 to CentOS 6.6)
>> >>> >
>> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
>> network
>> >>> > anymore.
>> >>> > And some VMs are down - and cant be started because they depend on
>> this
>> >>> > VR...
>> >>> >
>> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
>> this
>> >>> one
>> >>> > fro Guest network fails to start:
>> >>> >
>> >>> > Here you can see, after agent copies template from secondary
>> storage, to
>> >>> > primary local storage, it created base image, and backing file - so
>> >>> storage
>> >>> > setup seems completed.
>> >>> >
>> >>> > Than all out of sudden we have errors:
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Marcus,
Thanks a lot fot hint
True, I have the 0 as the value for some reason in database, for couple of
NICs....
select * from nics where ip4_address like "46.232%" and broadcast_uri =
"vlan://500" and default_nic = 0;
results: http://pastebin.com/rDAe2RY9
or down there...
This Techvee-FileServer server is already running (still not dead) and I
can see 1 NIC from UI...
Should I reset all of these to 1 ?
What is the purpose of this field default_nic = 0.
vlan://500 in my case limits results only to the network for this VR that
is having problems...
Any suggestions ?
"id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
"ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
"reserver_name" "reservation_id" "device_id" "update_time" "isolation_uri"
"ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
"ip6_cidr" "secondary_ip" "display_nic"
"2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
"46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
22:06:23" \N \N "0" "1"
"3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
"46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
14:45:47" \N \N "0" "1"
"3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
"46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
02:10:45" \N \N "0" "1"
"707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
"46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
13:38:52" \N \N "0" "1"
"1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
"46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
08:15:18" \N \N "0" "1"
"3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
"46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
11:41:11" \N \N "0" "1"
"3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
"46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
12:02:59" \N \N "0" "1"
"1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
"46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
08:16:12" \N \N "0" "1"
"1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
"46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
08:16:32" \N \N "0" "1"
"435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
"46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
22:25:44" \N \N "0" "1"
"2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
"46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 22:54:29"
"vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
"1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
"46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
09:06:56" \N \N "0" "1"
"2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
"46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11 14:32:00"
"vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
"2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
"46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
13:58:11" \N \N "0" "1"
"2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
"46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
18:46:10" \N \N "0" "1"
"1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
"46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
10:33:59" \N \N "0" "1"
"1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
"46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
14:54:11" \N \N "0" "1"
"204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
"46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16 12:25:41"
"vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
"3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
"46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
12:03:42" \N \N "0" "1"
"29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
"46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
01:11:18" \N \N "0" "1"
"25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
"46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
00:35:20" \N \N "0" "1"
"2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
"46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12 14:16:23"
"vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> router code, the null pointer indicates that there's no default nic
> for one of your guests "Techvee-FileServer". I'd guess that if you
> delete/move just that host you may be able to start the router, or at
> least get past this. You can also look into the db and see if you can
> find its nics in the cloud.nics table and see if there is in fact one
> marked default.
>
> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> > Looks like the issue is that null pointer in CreateDhcpEntry for
> > either "Techvee-FileServer" or the DHCP entry immediately after that.
> > It would suggest some inconsistent/unexpected data when creating a
> > DHCP entry for one of the guests serviced by this router. It's too bad
> > that one bad entry is fatal for the whole router.
> >
> >
> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com>
> wrote:
> >> Not really - we are painfully migrating stopped VMs, from VPS network
> >> (Guest Shared netwotk) to VPCs...
> >>
> >> MGMT server sends the STOP command to agent, even though the VM was
> never
> >> started, BUT the storage provisioning from template to volume is done...
> >>
> >> We are also looking into some external help as we speak...
> >>
> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >>
> >>> Hi,
> >>>
> >>> Have you managed to get to the bottom of this?
> >>>
> >>> --
> >>> Sent from the Delta quadrant using Borg technology!
> >>>
> >>> Nux!
> >>> www.nux.ro
> >>>
> >>> ----- Original Message -----
> >>> > From: "Andrija Panic" <an...@gmail.com>
> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >>> > Subject: [URGENT - HELP NEEDED]
> >>>
> >>> > Hi guys,
> >>> >
> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> >>> before
> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >>> >
> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> network
> >>> > anymore.
> >>> > And some VMs are down - and cant be started because they depend on
> this
> >>> > VR...
> >>> >
> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
> this
> >>> one
> >>> > fro Guest network fails to start:
> >>> >
> >>> > Here you can see, after agent copies template from secondary
> storage, to
> >>> > primary local storage, it created base image, and backing file - so
> >>> storage
> >>> > setup seems completed.
> >>> >
> >>> > Than all out of sudden we have errors:
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Marcus,
Thanks a lot fot hint
True, I have the 0 as the value for some reason in database, for couple of
NICs....
select * from nics where ip4_address like "46.232%" and broadcast_uri =
"vlan://500" and default_nic = 0;
results: http://pastebin.com/rDAe2RY9
or down there...
This Techvee-FileServer server is already running (still not dead) and I
can see 1 NIC from UI...
Should I reset all of these to 1 ?
What is the purpose of this field default_nic = 0.
vlan://500 in my case limits results only to the network for this VR that
is having problems...
Any suggestions ?
"id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" "gateway"
"ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy"
"reserver_name" "reservation_id" "device_id" "update_time" "isolation_uri"
"ip6_address" "default_nic" "vm_type" "created" "removed" "ip6_gateway"
"ip6_cidr" "secondary_ip" "display_nic"
"2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" "06:70:0a:00:00:ac"
"46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04
23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04
22:06:23" \N \N "0" "1"
"3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" "06:fc:c2:00:00:ad"
"46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03
15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03
14:45:47" \N \N "0" "1"
"3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" "06:d7:ac:00:00:ae"
"46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19
03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19
02:10:45" \N \N "0" "1"
"707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72"
"46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27
14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27
13:38:52" \N \N "0" "1"
"1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" "06:1d:90:00:00:72"
"46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23
08:15:18" \N \N "0" "1"
"3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" "06:33:ac:00:00:77"
"46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16
11:41:11" \N \N "0" "1"
"3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" "06:8a:ac:00:00:77"
"46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16
12:02:59" \N \N "0" "1"
"1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" "06:c2:c4:00:00:7a"
"46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23
08:16:12" \N \N "0" "1"
"1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" "06:1c:78:00:00:7c"
"46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23
08:16:32" \N \N "0" "1"
"435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e"
"46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13
23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13
22:25:44" \N \N "0" "1"
"2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" "06:be:24:00:00:8e"
"46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 22:54:29"
"vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1"
"1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" "06:cd:d6:00:00:97"
"46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02
11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02
09:06:56" \N \N "0" "1"
"2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" "06:6e:0c:00:00:9c"
"46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11 14:32:00"
"vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1"
"2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" "06:b8:b4:00:00:a1"
"46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02
14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02
13:58:11" \N \N "0" "1"
"2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" "06:f8:36:00:00:a7"
"46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21
19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21
18:46:10" \N \N "0" "1"
"1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" "06:ac:4e:00:00:54"
"46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23
10:33:59" \N \N "0" "1"
"1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" "06:46:d8:00:00:54"
"46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23
16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23
14:54:11" \N \N "0" "1"
"204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35"
"46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16 12:25:41"
"vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1"
"3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" "06:c9:e0:00:00:3a"
"46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16
13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16
12:03:42" \N \N "0" "1"
"29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a"
"46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11
01:11:18" \N \N "0" "1"
"25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b"
"46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "206"
"Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11
01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11
00:35:20" \N \N "0" "1"
"2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" "06:02:a8:00:00:e2"
"46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" "212"
"Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12 14:16:23"
"vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1"
On 16 March 2015 at 16:11, Marcus <sh...@gmail.com> wrote:
> FWIW, if your 4.3.2 is the same as what's in the source tree for the
> router code, the null pointer indicates that there's no default nic
> for one of your guests "Techvee-FileServer". I'd guess that if you
> delete/move just that host you may be able to start the router, or at
> least get past this. You can also look into the db and see if you can
> find its nics in the cloud.nics table and see if there is in fact one
> marked default.
>
> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> > Looks like the issue is that null pointer in CreateDhcpEntry for
> > either "Techvee-FileServer" or the DHCP entry immediately after that.
> > It would suggest some inconsistent/unexpected data when creating a
> > DHCP entry for one of the guests serviced by this router. It's too bad
> > that one bad entry is fatal for the whole router.
> >
> >
> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com>
> wrote:
> >> Not really - we are painfully migrating stopped VMs, from VPS network
> >> (Guest Shared netwotk) to VPCs...
> >>
> >> MGMT server sends the STOP command to agent, even though the VM was
> never
> >> started, BUT the storage provisioning from template to volume is done...
> >>
> >> We are also looking into some external help as we speak...
> >>
> >> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> >>
> >>> Hi,
> >>>
> >>> Have you managed to get to the bottom of this?
> >>>
> >>> --
> >>> Sent from the Delta quadrant using Borg technology!
> >>>
> >>> Nux!
> >>> www.nux.ro
> >>>
> >>> ----- Original Message -----
> >>> > From: "Andrija Panic" <an...@gmail.com>
> >>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> >>> > Sent: Sunday, 15 March, 2015 16:19:07
> >>> > Subject: [URGENT - HELP NEEDED]
> >>>
> >>> > Hi guys,
> >>> >
> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> >>> before
> >>> > that, from CentOS 6.5 to CentOS 6.6)
> >>> >
> >>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> network
> >>> > anymore.
> >>> > And some VMs are down - and cant be started because they depend on
> this
> >>> > VR...
> >>> >
> >>> > VPC VRs are created fine, so new VR for VPC are created fine, but
> this
> >>> one
> >>> > fro Guest network fails to start:
> >>> >
> >>> > Here you can see, after agent copies template from secondary
> storage, to
> >>> > primary local storage, it created base image, and backing file - so
> >>> storage
> >>> > setup seems completed.
> >>> >
> >>> > Than all out of sudden we have errors:
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
FWIW, if your 4.3.2 is the same as what's in the source tree for the
router code, the null pointer indicates that there's no default nic
for one of your guests "Techvee-FileServer". I'd guess that if you
delete/move just that host you may be able to start the router, or at
least get past this. You can also look into the db and see if you can
find its nics in the cloud.nics table and see if there is in fact one
marked default.
On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> Looks like the issue is that null pointer in CreateDhcpEntry for
> either "Techvee-FileServer" or the DHCP entry immediately after that.
> It would suggest some inconsistent/unexpected data when creating a
> DHCP entry for one of the guests serviced by this router. It's too bad
> that one bad entry is fatal for the whole router.
>
>
> On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com> wrote:
>> Not really - we are painfully migrating stopped VMs, from VPS network
>> (Guest Shared netwotk) to VPCs...
>>
>> MGMT server sends the STOP command to agent, even though the VM was never
>> started, BUT the storage provisioning from template to volume is done...
>>
>> We are also looking into some external help as we speak...
>>
>> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>>
>>> Hi,
>>>
>>> Have you managed to get to the bottom of this?
>>>
>>> --
>>> Sent from the Delta quadrant using Borg technology!
>>>
>>> Nux!
>>> www.nux.ro
>>>
>>> ----- Original Message -----
>>> > From: "Andrija Panic" <an...@gmail.com>
>>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>>> > Sent: Sunday, 15 March, 2015 16:19:07
>>> > Subject: [URGENT - HELP NEEDED]
>>>
>>> > Hi guys,
>>> >
>>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>>> before
>>> > that, from CentOS 6.5 to CentOS 6.6)
>>> >
>>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
>>> > anymore.
>>> > And some VMs are down - and cant be started because they depend on this
>>> > VR...
>>> >
>>> > VPC VRs are created fine, so new VR for VPC are created fine, but this
>>> one
>>> > fro Guest network fails to start:
>>> >
>>> > Here you can see, after agent copies template from secondary storage, to
>>> > primary local storage, it created base image, and backing file - so
>>> storage
>>> > setup seems completed.
>>> >
>>> > Than all out of sudden we have errors:
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
FWIW, if your 4.3.2 is the same as what's in the source tree for the
router code, the null pointer indicates that there's no default nic
for one of your guests "Techvee-FileServer". I'd guess that if you
delete/move just that host you may be able to start the router, or at
least get past this. You can also look into the db and see if you can
find its nics in the cloud.nics table and see if there is in fact one
marked default.
On Mon, Mar 16, 2015 at 8:05 AM, Marcus <sh...@gmail.com> wrote:
> Looks like the issue is that null pointer in CreateDhcpEntry for
> either "Techvee-FileServer" or the DHCP entry immediately after that.
> It would suggest some inconsistent/unexpected data when creating a
> DHCP entry for one of the guests serviced by this router. It's too bad
> that one bad entry is fatal for the whole router.
>
>
> On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com> wrote:
>> Not really - we are painfully migrating stopped VMs, from VPS network
>> (Guest Shared netwotk) to VPCs...
>>
>> MGMT server sends the STOP command to agent, even though the VM was never
>> started, BUT the storage provisioning from template to volume is done...
>>
>> We are also looking into some external help as we speak...
>>
>> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>>
>>> Hi,
>>>
>>> Have you managed to get to the bottom of this?
>>>
>>> --
>>> Sent from the Delta quadrant using Borg technology!
>>>
>>> Nux!
>>> www.nux.ro
>>>
>>> ----- Original Message -----
>>> > From: "Andrija Panic" <an...@gmail.com>
>>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>>> > Sent: Sunday, 15 March, 2015 16:19:07
>>> > Subject: [URGENT - HELP NEEDED]
>>>
>>> > Hi guys,
>>> >
>>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>>> before
>>> > that, from CentOS 6.5 to CentOS 6.6)
>>> >
>>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
>>> > anymore.
>>> > And some VMs are down - and cant be started because they depend on this
>>> > VR...
>>> >
>>> > VPC VRs are created fine, so new VR for VPC are created fine, but this
>>> one
>>> > fro Guest network fails to start:
>>> >
>>> > Here you can see, after agent copies template from secondary storage, to
>>> > primary local storage, it created base image, and backing file - so
>>> storage
>>> > setup seems completed.
>>> >
>>> > Than all out of sudden we have errors:
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Looks like the issue is that null pointer in CreateDhcpEntry for
either "Techvee-FileServer" or the DHCP entry immediately after that.
It would suggest some inconsistent/unexpected data when creating a
DHCP entry for one of the guests serviced by this router. It's too bad
that one bad entry is fatal for the whole router.
On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com> wrote:
> Not really - we are painfully migrating stopped VMs, from VPS network
> (Guest Shared netwotk) to VPCs...
>
> MGMT server sends the STOP command to agent, even though the VM was never
> started, BUT the storage provisioning from template to volume is done...
>
> We are also looking into some external help as we speak...
>
> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>
>> Hi,
>>
>> Have you managed to get to the bottom of this?
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>> > From: "Andrija Panic" <an...@gmail.com>
>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> > Sent: Sunday, 15 March, 2015 16:19:07
>> > Subject: [URGENT - HELP NEEDED]
>>
>> > Hi guys,
>> >
>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>> before
>> > that, from CentOS 6.5 to CentOS 6.6)
>> >
>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
>> > anymore.
>> > And some VMs are down - and cant be started because they depend on this
>> > VR...
>> >
>> > VPC VRs are created fine, so new VR for VPC are created fine, but this
>> one
>> > fro Guest network fails to start:
>> >
>> > Here you can see, after agent copies template from secondary storage, to
>> > primary local storage, it created base image, and backing file - so
>> storage
>> > setup seems completed.
>> >
>> > Than all out of sudden we have errors:
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Marcus <sh...@gmail.com>.
Looks like the issue is that null pointer in CreateDhcpEntry for
either "Techvee-FileServer" or the DHCP entry immediately after that.
It would suggest some inconsistent/unexpected data when creating a
DHCP entry for one of the guests serviced by this router. It's too bad
that one bad entry is fatal for the whole router.
On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic <an...@gmail.com> wrote:
> Not really - we are painfully migrating stopped VMs, from VPS network
> (Guest Shared netwotk) to VPCs...
>
> MGMT server sends the STOP command to agent, even though the VM was never
> started, BUT the storage provisioning from template to volume is done...
>
> We are also looking into some external help as we speak...
>
> On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
>
>> Hi,
>>
>> Have you managed to get to the bottom of this?
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>> > From: "Andrija Panic" <an...@gmail.com>
>> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
>> > Sent: Sunday, 15 March, 2015 16:19:07
>> > Subject: [URGENT - HELP NEEDED]
>>
>> > Hi guys,
>> >
>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
>> before
>> > that, from CentOS 6.5 to CentOS 6.6)
>> >
>> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
>> > anymore.
>> > And some VMs are down - and cant be started because they depend on this
>> > VR...
>> >
>> > VPC VRs are created fine, so new VR for VPC are created fine, but this
>> one
>> > fro Guest network fails to start:
>> >
>> > Here you can see, after agent copies template from secondary storage, to
>> > primary local storage, it created base image, and backing file - so
>> storage
>> > setup seems completed.
>> >
>> > Than all out of sudden we have errors:
>>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Not really - we are painfully migrating stopped VMs, from VPS network
(Guest Shared netwotk) to VPCs...
MGMT server sends the STOP command to agent, even though the VM was never
started, BUT the storage provisioning from template to volume is done...
We are also looking into some external help as we speak...
On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> Hi,
>
> Have you managed to get to the bottom of this?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> > Sent: Sunday, 15 March, 2015 16:19:07
> > Subject: [URGENT - HELP NEEDED]
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Not really - we are painfully migrating stopped VMs, from VPS network
(Guest Shared netwotk) to VPCs...
MGMT server sends the STOP command to agent, even though the VM was never
started, BUT the storage provisioning from template to volume is done...
We are also looking into some external help as we speak...
On 16 March 2015 at 14:52, Nux! <nu...@li.nux.ro> wrote:
> Hi,
>
> Have you managed to get to the bottom of this?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> > Sent: Sunday, 15 March, 2015 16:19:07
> > Subject: [URGENT - HELP NEEDED]
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Hi,
Have you managed to get to the bottom of this?
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> Sent: Sunday, 15 March, 2015 16:19:07
> Subject: [URGENT - HELP NEEDED]
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
I just manually addded .ssh/id_rsa.pub.cloud content, to the
authorized_keys on host 22 - can login now fine, but still doesnt work.
And this is NOT present on other hosts from what I can see...
On 15 March 2015 at 22:52, Andrija Panic <an...@gmail.com> wrote:
> Yes, the host 22 is the physical host where the VR should be starting,,,
> host is UP and green in UI.
>
> [root@cs1 ~]# ssh -i .ssh/id_rsa.cloud root@10.44.253.12 # this is host 22
> This does NOT work, asks for pass - but yes I can login via password...
>
> Should this work at all ? The public key is not present in the
> authorized_keys on the host 22
>
> thx
>
> On 15 March 2015 at 22:35, Paul Angus <pa...@shapeblue.com> wrote:
>
>> That's correct. So the next error is:
>>
>> Resource [Host:22] is unreachable: Host 22: Unable to start instance due
>> to null
>>
>> Can you confirm that you can ssh into host 22 using the root account,
>> probably worth checking if the host is 'Up' in the CloudStack UI.
>> Also check the cloudagent.log - make sure that the agent is running
>> (restarting the agent won't hurt - monitor the agent logs while you're
>> doing it).
>>
>> Regards,
>>
>> Paul Angus
>> Cloud Architect
>> D: +44 20 3468 5163 |S: +44 20 3603 0540 | M: +44 7711 418 784 | T:
>> @CloudyAngus
>> paul.angus@shapeblue.com
>>
>> -----Original Message-----
>> From: Andrija Panic [mailto:andrija.panic@gmail.com]
>> Sent: 15 March 2015 20:24
>> To: users@cloudstack.apache.org
>> Subject: Re: [URGENT - HELP NEEDED]
>>
>> Hi Paul,
>>
>> I just checked the keys - they seems to be ther both the private and
>> public key...but what I can see on this particular host (I tied the
>> domainrouter to particular host, and on local storage) to track the problem
>> more easy...
>>
>> and from virsh and libvirt logs, I can see there is no VM being started
>> at all, so the keys would be later the problem, if the VM was started and
>> so on ?
>>
>> Thanks,
>> Andrija
>>
>>
>> On 15 March 2015 at 21:15, Paul Angus <pa...@shapeblue.com> wrote:
>>
>> > Hi Andrija,
>> >
>> > I see the first error which appears is:
>> >
>> > 2015-03-15 20:25:37,504 ERROR [c.c.v.VirtualMachineManagerImpl]
>> > (Job-Executor-27:ctx-1df42382 ctx-42389d7b) Failed to start instance
>> > VM[DomainRouter|r-2288-VM] java.lang.NullPointerException
>> > at
>> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createDhcp
>> > EntryCommand(VirtualNetworkApplianceManagerImpl.java:3527)
>> >
>> > This usually means CloudStack is unable to communicate with the
>> > virtual router. For KVM the communication if is over the linklocal
>> network.
>> >
>> > Is the public key at /root/.ssh/id_rsa.cloud and is it consistent with
>> > the keys in any hosts where the connection to virtual router is
>> > behaving normally?
>> >
>> >
>> >
>> > Regards
>> >
>> > Paul Angus
>> > Cloud Architect
>> > S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
>> > paul.angus@shapeblue.com
>> >
>> > -----Original Message-----
>> > From: Andrija Panic [mailto:andrija.panic@gmail.com]
>> > Sent: 15 March 2015 19:36
>> > To: dev@cloudstack.apache.org
>> > Cc: users@cloudstack.apache.org
>> > Subject: Re: [URGENT - HELP NEEDED]
>> >
>> > Hi Laszlo,
>> >
>> > here is the mgmt log:
>> >
>> > http://pastebin.com/3Zt4EDfM
>> >
>> > I found a strange line in the middle of the long exception stuff:
>> > - at com.sun.proxy.$Proxy240.startRouter(Unknown Source) As far as I
>> > can see, by the mounting secondary storage xxxx/template/2/615/ folder
>> > in the agent logs - this is exactly the same template that is used to
>> > provision VR for VPC, which works fine...
>> > You will see the template has some strange download path (from agent
>> > log)
>> > - because I used shapeblue 4.3.2 with GHOST patch, but it was qemu
>> > version 3, and our qemu doesnt support it, so I converted it back to
>> > version2, and put on my webserver, from whih it is downloaded (as per
>> > http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerability
>> > stack/
>> > )
>> >
>> > Thanks,
>> >
>> >
>> >
>> > On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com>
>> > wrote:
>> >
>> > > Hi Andrija,
>> > >
>> > > Could you also share management server logs?
>> > >
>> > >
>> > > On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic
>> > > <an...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi guys,
>> > > >
>> > > > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
>> > > > right
>> > > before
>> > > > that, from CentOS 6.5 to CentOS 6.6)
>> > > >
>> > > > And now I can not start SYSTEM VR - that is used for SHARED GUEST
>> > > > network anymore.
>> > > > And some VMs are down - and cant be started because they depend on
>> > > > this VR...
>> > > >
>> > > > VPC VRs are created fine, so new VR for VPC are created fine, but
>> > > > this
>> > > one
>> > > > fro Guest network fails to start:
>> > > >
>> > > > Here you can see, after agent copies template from secondary
>> > > > storage, to primary local storage, it created base image, and
>> > > > backing file - so
>> > > storage
>> > > > setup seems completed.
>> > > >
>> > > > Than all out of sudden we have errors:
>> > > >
>> > > > can't find connection: KVM, for vm: r-2281-VM, continue
>> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
>> > > > r-2281-VM, continue
>> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
>> > > > used , then use the default hypervisor
>> > > > 2015-03-15 16:52:45,678 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
>> > > > org.libvirt.LibvirtException: Domain not found: no domain with
>> > > > matching name 'r-2281-VM'
>> > > >
>> > > > - and BTW - absolutely NOTHING in the libvirt log !
>> > > >
>> > > >
>> > > > Here is the full log, with all the steps, on the host that doesnt
>> > > > host
>> > > any
>> > > > other VMs (except one - so log is easy to read) - you can notice
>> > > > after
>> > > 1st
>> > > > fail, agent copies new image/backing file, and tries again with
>> > > > new image...
>> > > >
>> > > > And BTW, same template is of course used for both System VR and
>> > > > VPC
>> > > VRs...
>> > > >
>> > > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd ,
>> > MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
>> > > >
>> > > >
>> > >
>> >
>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
>> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>> > > >
>> > > >
>> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
>> > > CO
>> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
>> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.c
>> > > lo ud.agent.api.to.NfsTO":{"_url":"nfs://
>> > > > 10.44.253.13/var/lib/libvirt/SECONDARY
>> > > >
>> > > >
>> > >
>> >
>> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
>> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>> > > >
>> > > >
>> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
>> > > CO
>> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
>> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
>> > > pa
>> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
>> > > -4
>> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
>> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
>> > > //
>> > > >
>> > > >
>> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
>> > > e-
>> > > 5617-445e-983f-0885a744cab7
>> > > >
>> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
>> > > yp e":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
>> > > > }
>> > > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-4:null) Processing command:
>> > > > org.apache.cloudstack.storage.command.CopyCommand
>> > > > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) createStoragePool didn't find
>> > > > existing running pool: org.libvirt.LibvirtException: Storage pool
>> not found:
>> > > > no storage pool with matching uuid 'M2 N0 e , , need to create
>> > > > it
>> > > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) Didn't find an existing storage pool
>> > > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools
>> > > > with duplicate paths
>> > > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) Checking path of existing pool
>> > > > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to
>> > > > create
>> > > > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) Checking path of existing pool
>> > > > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to
>> > > > create
>> > > > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) Checking path of existing pool
>> > > > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to
>> > > > create
>> > > > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) Attempting to create storage pool
>> > > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
>> > > > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) <pool type='netfs'>
>> > > > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
>> > > > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
>> > > > <source>
>> > > > <host name='10.44.253.13'/>
>> > > > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
>> > > > </source>
>> > > > <target>
>> > > > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
>> > > > </target>
>> > > > </pool>
>> > > >
>> > > > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-4:null) <volume>
>> > > > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
>> > > > <capacity>2621440000</capacity>
>> > > > <target>
>> > > > <format type='qcow2'/>
>> > > > <permissions><mode>0744</mode></permissions></target>
>> > > > </volume>
>> > > >
>> > > > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-4:null) Executing: qemu-img info
>> > > >
>> > > >
>> > > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
>> > > f1
>> > > 9ac76b20.qcow2
>> > > > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-4:null) Execution is successful.
>> > > > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
>> > > >
>> > > >
>> > > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
>> > > f1
>> > > 9ac76b20.qcow2
>> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
>> > > > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-4:null) Execution is successful.
>> > > > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
>> > > >
>> > > >
>> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
>> > > "o
>> > > rg.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-
>> > > 7e
>> > > 8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm"
>> > > :false}},"result":true,"wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd ,
>> > MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
>> > > >
>> > > >
>> > >
>> >
>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
>> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>> > > >
>> > > >
>> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
>> > > CO
>> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
>> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
>> > > pa
>> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
>> > > -4
>> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
>> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
>> > > //
>> > > >
>> > > >
>> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
>> > > e-
>> > > 5617-445e-983f-0885a744cab7
>> > > >
>> > > >
>> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
>> > > yp
>> > > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
>> > > :{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT",
>> > > "d
>> > > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
>> > > ui
>> > > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
>> > > ys
>> > > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
>> > > url":"Filesystem://
>> > > >
>> > > >
>> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
>> > > e-
>> > > 5617-445e-983f-0885a744cab7
>> > > >
>> > > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r
>> > > -2
>> > > 281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hyper
>> > > vi
>> > > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-2:null) Processing command:
>> > > > org.apache.cloudstack.storage.command.CopyCommand
>> > > > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-2:null) <volume>
>> > > > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
>> > > > <capacity>2621440000</capacity>
>> > > > <target>
>> > > > <format type='qcow2'/>
>> > > > <permissions><mode>0744</mode></permissions></target>
>> > > > </volume>
>> > > >
>> > > > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2
>> > > > -b
>> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
>> > > > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
>> > > > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-2:null) Execution is successful.
>> > > > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
>> > > > environment variable not set. Disabling LTTng-UST per-user tracing.
>> > > > (in
>> > > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
>> > > '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
>> > > > fmt=qcow2 size=2621440000
>> > > >
>> > >
>> >
>> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
>> > > > encryption=off cluster_size=65536
>> > > >
>> > > > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
>> > > >
>> > > >
>> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
>> > > "o
>> > > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
>> > > pa
>> > > th":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"Q
>> > > CO W2","id":0}},"result":true,"wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd ,
>> > MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
>> > > >
>> > > >
>> > > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
>> > > nc e":false,"vmName":"r-2281-VM","wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-3:null) Processing command:
>> > > > com.cloud.agent.api.StopCommand
>> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
>> > > > r-2281-VM, continue
>> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
>> > > > r-2281-VM, continue
>> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
>> > > > used , then use the default hypervisor
>> > > > 2015-03-15 16:52:45,678 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
>> > > > org.libvirt.LibvirtException: Domain not found: no domain with
>> > > > matching name 'r-2281-VM'
>> > > > 2015-03-15 16:52:45,678 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
>> > > > org.libvirt.LibvirtException: Domain not found: no domain with
>> > > > matching name 'r-2281-VM'
>> > > > 2015-03-15 16:52:45,679 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-3:null) Try to stop the vm at first
>> > > > 2015-03-15 16:52:45,679 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need
>> > > > to stop it
>> > > > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
>> > > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>> > > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd ,
>> > MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
>> > > >
>> > > >
>> > >
>> >
>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
>> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>> > > >
>> > > >
>> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
>> > > CO
>> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
>> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
>> > > pa
>> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
>> > > -4
>> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
>> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
>> > > //
>> > > >
>> > > >
>> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
>> > > e-
>> > > 5617-445e-983f-0885a744cab7
>> > > >
>> > > >
>> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
>> > > yp
>> > > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
>> > > :{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT",
>> > > "d
>> > > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
>> > > ui
>> > > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
>> > > ys
>> > > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
>> > > url":"Filesystem://
>> > > >
>> > > >
>> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
>> > > e-
>> > > 5617-445e-983f-0885a744cab7
>> > > >
>> > > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r
>> > > -2
>> > > 281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hyper
>> > > vi
>> > > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-1:null) Processing command:
>> > > > org.apache.cloudstack.storage.command.CopyCommand
>> > > > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
>> > > > (agentRequest-Handler-1:null) <volume>
>> > > > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
>> > > > <capacity>2621440000</capacity>
>> > > > <target>
>> > > > <format type='qcow2'/>
>> > > > <permissions><mode>0744</mode></permissions></target>
>> > > > </volume>
>> > > >
>> > > > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2
>> > > > -b
>> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
>> > > > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
>> > > > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-1:null) Execution is successful.
>> > > > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
>> > > > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
>> > > > environment variable not set. Disabling LTTng-UST per-user tracing.
>> > > > (in
>> > > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
>> > > '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
>> > > > fmt=qcow2 size=2621440000
>> > > >
>> > >
>> >
>> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
>> > > > encryption=off cluster_size=65536
>> > > >
>> > > > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
>> > > >
>> > > >
>> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
>> > > "o
>> > > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
>> > > pa
>> > > th":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"Q
>> > > CO W2","id":0}},"result":true,"wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd ,
>> > MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
>> > > >
>> > > >
>> > > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
>> > > nc e":false,"vmName":"r-2281-VM","wait":0}}]
>> > > > }
>> > > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-5:null) Processing command:
>> > > > com.cloud.agent.api.StopCommand
>> > > > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
>> > > > r-2281-VM, continue
>> > > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
>> > > > r-2281-VM, continue
>> > > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
>> > > > (agentRequest-Handler-5:null) can't find which hypervisor the vm
>> > > > used , then use the default hypervisor
>> > > > 2015-03-15 16:52:46,694 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-5:null) Failed to get dom xml:
>> > > > org.libvirt.LibvirtException: Domain not found: no domain with
>> > > > matching name 'r-2281-VM'
>> > > > 2015-03-15 16:52:46,694 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-5:null) Failed to get dom xml:
>> > > > org.libvirt.LibvirtException: Domain not found: no domain with
>> > > > matching name 'r-2281-VM'
>> > > > 2015-03-15 16:52:46,694 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-5:null) Try to stop the vm at first
>> > > > 2015-03-15 16:52:46,695 DEBUG
>> > > > [kvm.resource.LibvirtComputingResource]
>> > > > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need
>> > > > to stop it
>> > > > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
>> > > > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
>> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
>> > > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Andrija Panić
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > EOF
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>> > Find out more about ShapeBlue and our range of CloudStack related
>> > services
>> >
>> > IaaS Cloud Design & Build<
>> > http://shapeblue.com/iaas-cloud-design-and-build//>
>> > CSForge – rapid IaaS deployment
>> > framework<http://shapeblue.com/csforge/>
>> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> > CloudStack Software Engineering<
>> > http://shapeblue.com/cloudstack-software-engineering/>
>> > CloudStack Infrastructure Support<
>> > http://shapeblue.com/cloudstack-infrastructure-support/>
>> > CloudStack Bootcamp Training Courses<
>> > http://shapeblue.com/cloudstack-training/>
>> >
>> > This email and any attachments to it may be confidential and are
>> > intended solely for the use of the individual to whom it is addressed.
>> > Any views or opinions expressed are solely those of the author and do
>> > not necessarily represent those of Shape Blue Ltd or related
>> > companies. If you are not the intended recipient of this email, you
>> > must neither take any action based upon its contents, nor copy or show
>> > it to anyone. Please contact the sender if you believe you have
>> > received this email in error. Shape Blue Ltd is a company incorporated
>> > in England & Wales. ShapeBlue Services India LLP is a company
>> > incorporated in India and is operated under license from Shape Blue
>> > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
>> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
>> > Pty Ltd is a company registered by The Republic of South Africa and is
>> traded under license from Shape Blue Ltd. ShapeBlue is a registered
>> trademark.
>> >
>>
>>
>>
>> --
>>
>> Andrija Panić
>> Find out more about ShapeBlue and our range of CloudStack related services
>>
>> IaaS Cloud Design & Build<
>> http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Software Engineering<
>> http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<
>> http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training Courses<
>> http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are intended
>> solely for the use of the individual to whom it is addressed. Any views or
>> opinions expressed are solely those of the author and do not necessarily
>> represent those of Shape Blue Ltd or related companies. If you are not the
>> intended recipient of this email, you must neither take any action based
>> upon its contents, nor copy or show it to anyone. Please contact the sender
>> if you believe you have received this email in error. Shape Blue Ltd is a
>> company incorporated in England & Wales. ShapeBlue Services India LLP is a
>> company incorporated in India and is operated under license from Shape Blue
>> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
>> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
>> a company registered by The Republic of South Africa and is traded under
>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Yes, the host 22 is the physical host where the VR should be starting,,,
host is UP and green in UI.
[root@cs1 ~]# ssh -i .ssh/id_rsa.cloud root@10.44.253.12 # this is host 22
This does NOT work, asks for pass - but yes I can login via password...
Should this work at all ? The public key is not present in the
authorized_keys on the host 22
thx
On 15 March 2015 at 22:35, Paul Angus <pa...@shapeblue.com> wrote:
> That's correct. So the next error is:
>
> Resource [Host:22] is unreachable: Host 22: Unable to start instance due
> to null
>
> Can you confirm that you can ssh into host 22 using the root account,
> probably worth checking if the host is 'Up' in the CloudStack UI.
> Also check the cloudagent.log - make sure that the agent is running
> (restarting the agent won't hurt - monitor the agent logs while you're
> doing it).
>
> Regards,
>
> Paul Angus
> Cloud Architect
> D: +44 20 3468 5163 |S: +44 20 3603 0540 | M: +44 7711 418 784 | T:
> @CloudyAngus
> paul.angus@shapeblue.com
>
> -----Original Message-----
> From: Andrija Panic [mailto:andrija.panic@gmail.com]
> Sent: 15 March 2015 20:24
> To: users@cloudstack.apache.org
> Subject: Re: [URGENT - HELP NEEDED]
>
> Hi Paul,
>
> I just checked the keys - they seems to be ther both the private and
> public key...but what I can see on this particular host (I tied the
> domainrouter to particular host, and on local storage) to track the problem
> more easy...
>
> and from virsh and libvirt logs, I can see there is no VM being started at
> all, so the keys would be later the problem, if the VM was started and so
> on ?
>
> Thanks,
> Andrija
>
>
> On 15 March 2015 at 21:15, Paul Angus <pa...@shapeblue.com> wrote:
>
> > Hi Andrija,
> >
> > I see the first error which appears is:
> >
> > 2015-03-15 20:25:37,504 ERROR [c.c.v.VirtualMachineManagerImpl]
> > (Job-Executor-27:ctx-1df42382 ctx-42389d7b) Failed to start instance
> > VM[DomainRouter|r-2288-VM] java.lang.NullPointerException
> > at
> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createDhcp
> > EntryCommand(VirtualNetworkApplianceManagerImpl.java:3527)
> >
> > This usually means CloudStack is unable to communicate with the
> > virtual router. For KVM the communication if is over the linklocal
> network.
> >
> > Is the public key at /root/.ssh/id_rsa.cloud and is it consistent with
> > the keys in any hosts where the connection to virtual router is
> > behaving normally?
> >
> >
> >
> > Regards
> >
> > Paul Angus
> > Cloud Architect
> > S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
> > paul.angus@shapeblue.com
> >
> > -----Original Message-----
> > From: Andrija Panic [mailto:andrija.panic@gmail.com]
> > Sent: 15 March 2015 19:36
> > To: dev@cloudstack.apache.org
> > Cc: users@cloudstack.apache.org
> > Subject: Re: [URGENT - HELP NEEDED]
> >
> > Hi Laszlo,
> >
> > here is the mgmt log:
> >
> > http://pastebin.com/3Zt4EDfM
> >
> > I found a strange line in the middle of the long exception stuff:
> > - at com.sun.proxy.$Proxy240.startRouter(Unknown Source) As far as I
> > can see, by the mounting secondary storage xxxx/template/2/615/ folder
> > in the agent logs - this is exactly the same template that is used to
> > provision VR for VPC, which works fine...
> > You will see the template has some strange download path (from agent
> > log)
> > - because I used shapeblue 4.3.2 with GHOST patch, but it was qemu
> > version 3, and our qemu doesnt support it, so I converted it back to
> > version2, and put on my webserver, from whih it is downloaded (as per
> > http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerability
> > stack/
> > )
> >
> > Thanks,
> >
> >
> >
> > On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com>
> > wrote:
> >
> > > Hi Andrija,
> > >
> > > Could you also share management server logs?
> > >
> > >
> > > On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic
> > > <an...@gmail.com>
> > > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> > > > right
> > > before
> > > > that, from CentOS 6.5 to CentOS 6.6)
> > > >
> > > > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> > > > network anymore.
> > > > And some VMs are down - and cant be started because they depend on
> > > > this VR...
> > > >
> > > > VPC VRs are created fine, so new VR for VPC are created fine, but
> > > > this
> > > one
> > > > fro Guest network fails to start:
> > > >
> > > > Here you can see, after agent copies template from secondary
> > > > storage, to primary local storage, it created base image, and
> > > > backing file - so
> > > storage
> > > > setup seems completed.
> > > >
> > > > Than all out of sudden we have errors:
> > > >
> > > > can't find connection: KVM, for vm: r-2281-VM, continue
> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > > r-2281-VM, continue
> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > > used , then use the default hypervisor
> > > > 2015-03-15 16:52:45,678 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > > matching name 'r-2281-VM'
> > > >
> > > > - and BTW - absolutely NOTHING in the libvirt log !
> > > >
> > > >
> > > > Here is the full log, with all the steps, on the host that doesnt
> > > > host
> > > any
> > > > other VMs (except one - so log is easy to read) - you can notice
> > > > after
> > > 1st
> > > > fail, agent copies new image/backing file, and tries again with
> > > > new image...
> > > >
> > > > And BTW, same template is of course used for both System VR and
> > > > VPC
> > > VRs...
> > > >
> > > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd ,
> > MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > > >
> > > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > > >
> > > >
> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > > CO
> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.c
> > > lo ud.agent.api.to.NfsTO":{"_url":"nfs://
> > > > 10.44.253.13/var/lib/libvirt/SECONDARY
> > > >
> > > >
> > >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > > >
> > > >
> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > > CO
> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > > pa
> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > > -4
> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > > //
> > > >
> > > >
> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > > e-
> > > 5617-445e-983f-0885a744cab7
> > > >
> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > > yp e":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > > > }
> > > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-4:null) Processing command:
> > > > org.apache.cloudstack.storage.command.CopyCommand
> > > > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) createStoragePool didn't find
> > > > existing running pool: org.libvirt.LibvirtException: Storage pool
> not found:
> > > > no storage pool with matching uuid 'M2 N0 e , , need to create
> > > > it
> > > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools
> > > > with duplicate paths
> > > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to
> > > > create
> > > > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to
> > > > create
> > > > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to
> > > > create
> > > > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) Attempting to create storage pool
> > > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > > > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) <pool type='netfs'>
> > > > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > > > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > > > <source>
> > > > <host name='10.44.253.13'/>
> > > > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > > > </source>
> > > > <target>
> > > > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > > > </target>
> > > > </pool>
> > > >
> > > > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-4:null) <volume>
> > > > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > > > <capacity>2621440000</capacity>
> > > > <target>
> > > > <format type='qcow2'/>
> > > > <permissions><mode>0744</mode></permissions></target>
> > > > </volume>
> > > >
> > > > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-4:null) Executing: qemu-img info
> > > >
> > > >
> > > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
> > > f1
> > > 9ac76b20.qcow2
> > > > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-4:null) Execution is successful.
> > > > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> > > >
> > > >
> > > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
> > > f1
> > > 9ac76b20.qcow2
> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-4:null) Execution is successful.
> > > > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > >
> > > >
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > > "o
> > > rg.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-
> > > 7e
> > > 8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm"
> > > :false}},"result":true,"wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd ,
> > MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > > >
> > > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > > >
> > > >
> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > > CO
> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > > pa
> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > > -4
> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > > //
> > > >
> > > >
> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > > e-
> > > 5617-445e-983f-0885a744cab7
> > > >
> > > >
> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > > yp
> > > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > > :{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT",
> > > "d
> > > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
> > > ui
> > > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
> > > ys
> > > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > > url":"Filesystem://
> > > >
> > > >
> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > > e-
> > > 5617-445e-983f-0885a744cab7
> > > >
> > > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r
> > > -2
> > > 281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hyper
> > > vi
> > > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-2:null) Processing command:
> > > > org.apache.cloudstack.storage.command.CopyCommand
> > > > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-2:null) <volume>
> > > > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > > > <capacity>2621440000</capacity>
> > > > <target>
> > > > <format type='qcow2'/>
> > > > <permissions><mode>0744</mode></permissions></target>
> > > > </volume>
> > > >
> > > > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2
> > > > -b
> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > > > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-2:null) Execution is successful.
> > > > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > > (in
> > > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > > '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > > > fmt=qcow2 size=2621440000
> > > >
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > > encryption=off cluster_size=65536
> > > >
> > > > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > >
> > > >
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > > "o
> > > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
> > > pa
> > > th":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"Q
> > > CO W2","id":0}},"result":true,"wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd ,
> > MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > > >
> > > >
> > > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
> > > nc e":false,"vmName":"r-2281-VM","wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-3:null) Processing command:
> > > > com.cloud.agent.api.StopCommand
> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > > > r-2281-VM, continue
> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > > r-2281-VM, continue
> > > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > > used , then use the default hypervisor
> > > > 2015-03-15 16:52:45,678 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > > matching name 'r-2281-VM'
> > > > 2015-03-15 16:52:45,678 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > > matching name 'r-2281-VM'
> > > > 2015-03-15 16:52:45,679 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-3:null) Try to stop the vm at first
> > > > 2015-03-15 16:52:45,679 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need
> > > > to stop it
> > > > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd ,
> > MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > > >
> > > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > > >
> > > >
> > > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > > CO
> > > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > > pa
> > > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > > -4
> > > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > > //
> > > >
> > > >
> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > > e-
> > > 5617-445e-983f-0885a744cab7
> > > >
> > > >
> > > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > > yp
> > > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > > :{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT",
> > > "d
> > > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
> > > ui
> > > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
> > > ys
> > > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > > url":"Filesystem://
> > > >
> > > >
> > > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > > e-
> > > 5617-445e-983f-0885a744cab7
> > > >
> > > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r
> > > -2
> > > 281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hyper
> > > vi
> > > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-1:null) Processing command:
> > > > org.apache.cloudstack.storage.command.CopyCommand
> > > > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > > (agentRequest-Handler-1:null) <volume>
> > > > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > > > <capacity>2621440000</capacity>
> > > > <target>
> > > > <format type='qcow2'/>
> > > > <permissions><mode>0744</mode></permissions></target>
> > > > </volume>
> > > >
> > > > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2
> > > > -b
> > > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > > > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-1:null) Execution is successful.
> > > > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > > > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > > (in
> > > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > > '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > > > fmt=qcow2 size=2621440000
> > > >
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > > encryption=off cluster_size=65536
> > > >
> > > > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > >
> > > >
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > > "o
> > > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
> > > pa
> > > th":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"Q
> > > CO W2","id":0}},"result":true,"wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd ,
> > MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > > >
> > > >
> > > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
> > > nc e":false,"vmName":"r-2281-VM","wait":0}}]
> > > > }
> > > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-5:null) Processing command:
> > > > com.cloud.agent.api.StopCommand
> > > > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > > > r-2281-VM, continue
> > > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > > > r-2281-VM, continue
> > > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > > (agentRequest-Handler-5:null) can't find which hypervisor the vm
> > > > used , then use the default hypervisor
> > > > 2015-03-15 16:52:46,694 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > > matching name 'r-2281-VM'
> > > > 2015-03-15 16:52:46,694 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > > matching name 'r-2281-VM'
> > > > 2015-03-15 16:52:46,694 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-5:null) Try to stop the vm at first
> > > > 2015-03-15 16:52:46,695 DEBUG
> > > > [kvm.resource.LibvirtComputingResource]
> > > > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need
> > > > to stop it
> > > > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > EOF
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> > Find out more about ShapeBlue and our range of CloudStack related
> > services
> >
> > IaaS Cloud Design & Build<
> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > CSForge – rapid IaaS deployment
> > framework<http://shapeblue.com/csforge/>
> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> > CloudStack Software Engineering<
> > http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support<
> > http://shapeblue.com/cloudstack-infrastructure-support/>
> > CloudStack Bootcamp Training Courses<
> > http://shapeblue.com/cloudstack-training/>
> >
> > This email and any attachments to it may be confidential and are
> > intended solely for the use of the individual to whom it is addressed.
> > Any views or opinions expressed are solely those of the author and do
> > not necessarily represent those of Shape Blue Ltd or related
> > companies. If you are not the intended recipient of this email, you
> > must neither take any action based upon its contents, nor copy or show
> > it to anyone. Please contact the sender if you believe you have
> > received this email in error. Shape Blue Ltd is a company incorporated
> > in England & Wales. ShapeBlue Services India LLP is a company
> > incorporated in India and is operated under license from Shape Blue
> > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
> > Pty Ltd is a company registered by The Republic of South Africa and is
> traded under license from Shape Blue Ltd. ShapeBlue is a registered
> trademark.
> >
>
>
>
> --
>
> Andrija Panić
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
--
Andrija Panić
RE: [URGENT - HELP NEEDED]
Posted by Paul Angus <pa...@shapeblue.com>.
That's correct. So the next error is:
Resource [Host:22] is unreachable: Host 22: Unable to start instance due to null
Can you confirm that you can ssh into host 22 using the root account, probably worth checking if the host is 'Up' in the CloudStack UI.
Also check the cloudagent.log - make sure that the agent is running (restarting the agent won't hurt - monitor the agent logs while you're doing it).
Regards,
Paul Angus
Cloud Architect
D: +44 20 3468 5163 |S: +44 20 3603 0540 | M: +44 7711 418 784 | T: @CloudyAngus
paul.angus@shapeblue.com
-----Original Message-----
From: Andrija Panic [mailto:andrija.panic@gmail.com]
Sent: 15 March 2015 20:24
To: users@cloudstack.apache.org
Subject: Re: [URGENT - HELP NEEDED]
Hi Paul,
I just checked the keys - they seems to be ther both the private and public key...but what I can see on this particular host (I tied the domainrouter to particular host, and on local storage) to track the problem more easy...
and from virsh and libvirt logs, I can see there is no VM being started at all, so the keys would be later the problem, if the VM was started and so on ?
Thanks,
Andrija
On 15 March 2015 at 21:15, Paul Angus <pa...@shapeblue.com> wrote:
> Hi Andrija,
>
> I see the first error which appears is:
>
> 2015-03-15 20:25:37,504 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Job-Executor-27:ctx-1df42382 ctx-42389d7b) Failed to start instance
> VM[DomainRouter|r-2288-VM] java.lang.NullPointerException
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createDhcp
> EntryCommand(VirtualNetworkApplianceManagerImpl.java:3527)
>
> This usually means CloudStack is unable to communicate with the
> virtual router. For KVM the communication if is over the linklocal network.
>
> Is the public key at /root/.ssh/id_rsa.cloud and is it consistent with
> the keys in any hosts where the connection to virtual router is
> behaving normally?
>
>
>
> Regards
>
> Paul Angus
> Cloud Architect
> S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
> paul.angus@shapeblue.com
>
> -----Original Message-----
> From: Andrija Panic [mailto:andrija.panic@gmail.com]
> Sent: 15 March 2015 19:36
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: [URGENT - HELP NEEDED]
>
> Hi Laszlo,
>
> here is the mgmt log:
>
> http://pastebin.com/3Zt4EDfM
>
> I found a strange line in the middle of the long exception stuff:
> - at com.sun.proxy.$Proxy240.startRouter(Unknown Source) As far as I
> can see, by the mounting secondary storage xxxx/template/2/615/ folder
> in the agent logs - this is exactly the same template that is used to
> provision VR for VPC, which works fine...
> You will see the template has some strange download path (from agent
> log)
> - because I used shapeblue 4.3.2 with GHOST patch, but it was qemu
> version 3, and our qemu doesnt support it, so I converted it back to
> version2, and put on my webserver, from whih it is downloaded (as per
> http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerability
> stack/
> )
>
> Thanks,
>
>
>
> On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com>
> wrote:
>
> > Hi Andrija,
> >
> > Could you also share management server logs?
> >
> >
> > On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic
> > <an...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated
> > > right
> > before
> > > that, from CentOS 6.5 to CentOS 6.6)
> > >
> > > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> > > network anymore.
> > > And some VMs are down - and cant be started because they depend on
> > > this VR...
> > >
> > > VPC VRs are created fine, so new VR for VPC are created fine, but
> > > this
> > one
> > > fro Guest network fails to start:
> > >
> > > Here you can see, after agent copies template from secondary
> > > storage, to primary local storage, it created base image, and
> > > backing file - so
> > storage
> > > setup seems completed.
> > >
> > > Than all out of sudden we have errors:
> > >
> > > can't find connection: KVM, for vm: r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > >
> > > - and BTW - absolutely NOTHING in the libvirt log !
> > >
> > >
> > > Here is the full log, with all the steps, on the host that doesnt
> > > host
> > any
> > > other VMs (except one - so log is easy to read) - you can notice
> > > after
> > 1st
> > > fail, agent copies new image/backing file, and tries again with
> > > new image...
> > >
> > > And BTW, same template is of course used for both System VR and
> > > VPC
> > VRs...
> > >
> > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > CO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.c
> > lo ud.agent.api.to.NfsTO":{"_url":"nfs://
> > > 10.44.253.13/var/lib/libvirt/SECONDARY
> > >
> > >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > CO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > pa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > -4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > //
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > yp e":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > > }
> > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) createStoragePool didn't find
> > > existing running pool: org.libvirt.LibvirtException: Storage pool not found:
> > > no storage pool with matching uuid 'M2 N0 e , , need to create
> > > it
> > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools
> > > with duplicate paths
> > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to
> > > create
> > > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to
> > > create
> > > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to
> > > create
> > > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Attempting to create storage pool
> > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) <pool type='netfs'>
> > > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > > <source>
> > > <host name='10.44.253.13'/>
> > > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > > </source>
> > > <target>
> > > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > > </target>
> > > </pool>
> > >
> > > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) <volume>
> > > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Executing: qemu-img info
> > >
> > >
> > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
> > f1
> > 9ac76b20.qcow2
> > > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Execution is successful.
> > > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> > >
> > >
> > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5
> > f1
> > 9ac76b20.qcow2
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Execution is successful.
> > > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > "o
> > rg.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-
> > 7e
> > 8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm"
> > :false}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > CO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > pa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > -4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > //
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > e-
> > 5617-445e-983f-0885a744cab7
> > >
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > yp
> > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > :{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT",
> > "d
> > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
> > ui
> > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
> > ys
> > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r
> > -2
> > 281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hyper
> > vi
> > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-2:null) <volume>
> > > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2
> > > -b
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) Execution is successful.
> > > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > (in
> > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > > fmt=qcow2 size=2621440000
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > encryption=off cluster_size=65536
> > >
> > > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > "o
> > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
> > pa
> > th":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"Q
> > CO W2","id":0}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
> > nc e":false,"vmName":"r-2281-VM","wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Processing command:
> > > com.cloud.agent.api.StopCommand
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:45,679 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Try to stop the vm at first
> > > 2015-03-15 16:52:45,679 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need
> > > to stop it
> > > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"Q
> > CO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.a
> > pa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617
> > -4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:
> > //
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > e-
> > 5617-445e-983f-0885a744cab7
> > >
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorT
> > yp
> > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > :{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT",
> > "d
> > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u
> > ui
> > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Files
> > ys
> > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9
> > e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r
> > -2
> > 281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hyper
> > vi
> > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-1:null) <volume>
> > > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2
> > > -b
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) Execution is successful.
> > > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > (in
> > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > > fmt=qcow2 size=2621440000
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > encryption=off cluster_size=65536
> > >
> > > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> > "o
> > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"
> > pa
> > th":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"Q
> > CO W2","id":0}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSeque
> > nc e":false,"vmName":"r-2281-VM","wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Processing command:
> > > com.cloud.agent.api.StopCommand
> > > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Try to stop the vm at first
> > > 2015-03-15 16:52:46,695 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need
> > > to stop it
> > > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > EOF
> >
>
>
>
> --
>
> Andrija Panić
> Find out more about ShapeBlue and our range of CloudStack related
> services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment
> framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related
> companies. If you are not the intended recipient of this email, you
> must neither take any action based upon its contents, nor copy or show
> it to anyone. Please contact the sender if you believe you have
> received this email in error. Shape Blue Ltd is a company incorporated
> in England & Wales. ShapeBlue Services India LLP is a company
> incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
> Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
--
Andrija Panić
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Paul,
I just checked the keys - they seems to be ther both the private and public
key...but what I can see on this particular host (I tied the domainrouter
to particular host, and on local storage) to track the problem more easy...
and from virsh and libvirt logs, I can see there is no VM being started at
all, so the keys would be later the problem, if the VM was started and so
on ?
Thanks,
Andrija
On 15 March 2015 at 21:15, Paul Angus <pa...@shapeblue.com> wrote:
> Hi Andrija,
>
> I see the first error which appears is:
>
> 2015-03-15 20:25:37,504 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Job-Executor-27:ctx-1df42382 ctx-42389d7b) Failed to start instance
> VM[DomainRouter|r-2288-VM]
> java.lang.NullPointerException
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createDhcpEntryCommand(VirtualNetworkApplianceManagerImpl.java:3527)
>
> This usually means CloudStack is unable to communicate with the virtual
> router. For KVM the communication if is over the linklocal network.
>
> Is the public key at /root/.ssh/id_rsa.cloud and is it consistent with the
> keys in any hosts where the connection to virtual router is behaving
> normally?
>
>
>
> Regards
>
> Paul Angus
> Cloud Architect
> S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
> paul.angus@shapeblue.com
>
> -----Original Message-----
> From: Andrija Panic [mailto:andrija.panic@gmail.com]
> Sent: 15 March 2015 19:36
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: [URGENT - HELP NEEDED]
>
> Hi Laszlo,
>
> here is the mgmt log:
>
> http://pastebin.com/3Zt4EDfM
>
> I found a strange line in the middle of the long exception stuff:
> - at com.sun.proxy.$Proxy240.startRouter(Unknown Source) As far as I can
> see, by the mounting secondary storage xxxx/template/2/615/ folder in the
> agent logs - this is exactly the same template that is used to provision VR
> for VPC, which works fine...
> You will see the template has some strange download path (from agent log)
> - because I used shapeblue 4.3.2 with GHOST patch, but it was qemu version
> 3, and our qemu doesnt support it, so I converted it back to version2, and
> put on my webserver, from whih it is downloaded (as per
> http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerabilitystack/
> )
>
> Thanks,
>
>
>
> On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com>
> wrote:
>
> > Hi Andrija,
> >
> > Could you also share management server logs?
> >
> >
> > On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic
> > <an...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> > before
> > > that, from CentOS 6.5 to CentOS 6.6)
> > >
> > > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> > > network anymore.
> > > And some VMs are down - and cant be started because they depend on
> > > this VR...
> > >
> > > VPC VRs are created fine, so new VR for VPC are created fine, but
> > > this
> > one
> > > fro Guest network fails to start:
> > >
> > > Here you can see, after agent copies template from secondary
> > > storage, to primary local storage, it created base image, and
> > > backing file - so
> > storage
> > > setup seems completed.
> > >
> > > Than all out of sudden we have errors:
> > >
> > > can't find connection: KVM, for vm: r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > >
> > > - and BTW - absolutely NOTHING in the libvirt log !
> > >
> > >
> > > Here is the full log, with all the steps, on the host that doesnt
> > > host
> > any
> > > other VMs (except one - so log is easy to read) - you can notice
> > > after
> > 1st
> > > fail, agent copies new image/backing file, and tries again with new
> > > image...
> > >
> > > And BTW, same template is of course used for both System VR and VPC
> > VRs...
> > >
> > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.clo
> > ud.agent.api.to.NfsTO":{"_url":"nfs://
> > > 10.44.253.13/var/lib/libvirt/SECONDARY
> > >
> > >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> > e":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > > }
> > > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > > running pool: org.libvirt.LibvirtException: Storage pool not found:
> > > no storage pool with matching uuid 'M2 N0 e , , need to create
> > > it
> > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools
> > > with duplicate paths
> > > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Checking path of existing pool
> > > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) Attempting to create storage pool
> > > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) <pool type='netfs'>
> > > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > > <source>
> > > <host name='10.44.253.13'/>
> > > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > > </source>
> > > <target>
> > > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > > </target>
> > > </pool>
> > >
> > > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-4:null) <volume>
> > > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Executing: qemu-img info
> > >
> > >
> > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f1
> > 9ac76b20.qcow2
> > > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Execution is successful.
> > > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> > >
> > >
> > /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f1
> > 9ac76b20.qcow2
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-4:null) Execution is successful.
> > > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> > rg.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e
> > 8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm"
> > :false}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> > 5617-445e-983f-0885a744cab7
> > >
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > :{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","d
> > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uui
> > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesys
> > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2
> > 281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervi
> > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-2:null) <volume>
> > > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) Execution is successful.
> > > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > (in
> > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > > fmt=qcow2 size=2621440000
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > encryption=off cluster_size=65536
> > >
> > > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"pa
> > th":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCO
> > W2","id":0}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequenc
> > e":false,"vmName":"r-2281-VM","wait":0}}]
> > > }
> > > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Processing command:
> > > com.cloud.agent.api.StopCommand
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:45,678 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:45,679 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) Try to stop the vm at first
> > > 2015-03-15 16:52:45,679 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to
> > > stop it
> > > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> > >
> > >
> > ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> > W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> > true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> > che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> > 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> > 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> > 5617-445e-983f-0885a744cab7
> > >
> > >
> > "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> > e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> > :{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","d
> > ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uui
> > d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesys
> > tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> > url":"Filesystem://
> > >
> > >
> > 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> > 5617-445e-983f-0885a744cab7
> > >
> > "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2
> > 281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervi
> > sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Processing command:
> > > org.apache.cloudstack.storage.command.CopyCommand
> > > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > > (agentRequest-Handler-1:null) <volume>
> > > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > > <capacity>2621440000</capacity>
> > > <target>
> > > <format type='qcow2'/>
> > > <permissions><mode>0744</mode></permissions></target>
> > > </volume>
> > >
> > > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) Execution is successful.
> > > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > > environment variable not set. Disabling LTTng-UST per-user tracing.
> > > (in
> > > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> > '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > > fmt=qcow2 size=2621440000
> > >
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > > encryption=off cluster_size=65536
> > >
> > > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > >
> > >
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> > rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"pa
> > th":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCO
> > W2","id":0}},"result":true,"wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd ,
> MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 100011,
> > >
> > >
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequenc
> > e":false,"vmName":"r-2281-VM","wait":0}}]
> > > }
> > > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Processing command:
> > > com.cloud.agent.api.StopCommand
> > > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > > r-2281-VM, continue
> > > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > > (agentRequest-Handler-5:null) can't find which hypervisor the vm
> > > used , then use the default hypervisor
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Failed to get dom xml:
> > > org.libvirt.LibvirtException: Domain not found: no domain with
> > > matching name 'r-2281-VM'
> > > 2015-03-15 16:52:46,694 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) Try to stop the vm at first
> > > 2015-03-15 16:52:46,695 DEBUG
> > > [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to
> > > stop it
> > > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > > 161344838950, via: 1, Ver: v1, Flags: 10,
> > > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > EOF
> >
>
>
>
> --
>
> Andrija Panić
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
--
Andrija Panić
RE: [URGENT - HELP NEEDED]
Posted by Paul Angus <pa...@shapeblue.com>.
Hi Andrija,
I see the first error which appears is:
2015-03-15 20:25:37,504 ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-27:ctx-1df42382 ctx-42389d7b) Failed to start instance VM[DomainRouter|r-2288-VM]
java.lang.NullPointerException
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createDhcpEntryCommand(VirtualNetworkApplianceManagerImpl.java:3527)
This usually means CloudStack is unable to communicate with the virtual router. For KVM the communication if is over the linklocal network.
Is the public key at /root/.ssh/id_rsa.cloud and is it consistent with the keys in any hosts where the connection to virtual router is behaving normally?
Regards
Paul Angus
Cloud Architect
S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
paul.angus@shapeblue.com
-----Original Message-----
From: Andrija Panic [mailto:andrija.panic@gmail.com]
Sent: 15 March 2015 19:36
To: dev@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: Re: [URGENT - HELP NEEDED]
Hi Laszlo,
here is the mgmt log:
http://pastebin.com/3Zt4EDfM
I found a strange line in the middle of the long exception stuff:
- at com.sun.proxy.$Proxy240.startRouter(Unknown Source) As far as I can see, by the mounting secondary storage xxxx/template/2/615/ folder in the agent logs - this is exactly the same template that is used to provision VR for VPC, which works fine...
You will see the template has some strange download path (from agent log) - because I used shapeblue 4.3.2 with GHOST patch, but it was qemu version 3, and our qemu doesnt support it, so I converted it back to version2, and put on my webserver, from whih it is downloaded (as per http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerabilitystack/
)
Thanks,
On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com> wrote:
> Hi Andrija,
>
> Could you also share management server logs?
>
>
> On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic
> <an...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST
> > network anymore.
> > And some VMs are down - and cant be started because they depend on
> > this VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but
> > this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary
> > storage, to primary local storage, it created base image, and
> > backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > used , then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with
> > matching name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt
> > host
> any
> > other VMs (except one - so log is easy to read) - you can notice
> > after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> > image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.clo
> ud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> 5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> e":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found:
> > no storage pool with matching uuid 'M2 N0 e , , need to create
> > it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools
> > with duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f1
> 9ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f1
> 9ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> rg.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e
> 8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm"
> :false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> 5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> :{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","d
> ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uui
> d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesys
> tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> 5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2
> 281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervi
> sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing.
> > (in
> > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"pa
> th":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCO
> W2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequenc
> e":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm
> > used , then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with
> > matching name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with
> > matching name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to
> > stop it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCO
> W2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":
> true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apa
> che.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-4
> 45e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.
> 253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> 5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorTyp
> e":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO"
> :{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","d
> ataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uui
> d":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesys
> tem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"
> url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-
> 5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2
> 281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervi
> sorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing.
> > (in
> > setup_local_apps() at lttng-ust-comm.c:305) Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"o
> rg.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"pa
> th":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCO
> W2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequenc
> e":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm
> > used , then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with
> > matching name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with
> > matching name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG
> > [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to
> > stop it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> EOF
>
--
Andrija Panić
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Laszlo,
here is the mgmt log:
http://pastebin.com/3Zt4EDfM
I found a strange line in the middle of the long exception stuff:
- at com.sun.proxy.$Proxy240.startRouter(Unknown Source)
As far as I can see, by the mounting secondary storage xxxx/template/2/615/
folder in the agent logs - this is exactly the same template that is used
to provision VR for VPC, which works fine...
You will see the template has some strange download path (from agent log) -
because I used shapeblue 4.3.2 with GHOST patch, but it was qemu version 3,
and our qemu doesnt support it, so I converted it back to version2, and put
on my webserver, from whih it is downloaded (as per
http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerabilitystack/
)
Thanks,
On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com> wrote:
> Hi Andrija,
>
> Could you also share management server logs?
>
>
> On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt host
> any
> > other VMs (except one - so log is easy to read) - you can notice after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> > image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found: no
> > storage pool with matching uuid 'M2��N0�e���,�, need to create it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> > duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop
> > it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop
> > it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> EOF
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Andrija Panic <an...@gmail.com>.
Hi Laszlo,
here is the mgmt log:
http://pastebin.com/3Zt4EDfM
I found a strange line in the middle of the long exception stuff:
- at com.sun.proxy.$Proxy240.startRouter(Unknown Source)
As far as I can see, by the mounting secondary storage xxxx/template/2/615/
folder in the agent logs - this is exactly the same template that is used
to provision VR for VPC, which works fine...
You will see the template has some strange download path (from agent log) -
because I used shapeblue 4.3.2 with GHOST patch, but it was qemu version 3,
and our qemu doesnt support it, so I converted it back to version2, and put
on my webserver, from whih it is downloaded (as per
http://shapeblue.com/cloudstack/cloudstack-and-the-ghost-vulnerabilitystack/
)
Thanks,
On 15 March 2015 at 20:23, Laszlo Hornyak <la...@gmail.com> wrote:
> Hi Andrija,
>
> Could you also share management server logs?
>
>
> On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt host
> any
> > other VMs (except one - so log is easy to read) - you can notice after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> > image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found: no
> > storage pool with matching uuid 'M2��N0�e���,�, need to create it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> > duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop
> > it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop
> > it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> EOF
>
--
Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Hi Andrija,
Could you also share management server logs?
On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
>
> can't find connection: KVM, for vm: r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
>
> - and BTW - absolutely NOTHING in the libvirt log !
>
>
> Here is the full log, with all the steps, on the host that doesnt host any
> other VMs (except one - so log is easy to read) - you can notice after 1st
> fail, agent copies new image/backing file, and tries again with new
> image...
>
> And BTW, same template is of course used for both System VR and VPC VRs...
>
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.44.253.13/var/lib/libvirt/SECONDARY
>
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> }
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid 'M2��N0�e���,�, need to create it
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Didn't find an existing storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> duplicate paths
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> <source>
> <host name='10.44.253.13'/>
> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> </source>
> <target>
> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> </target>
> </pool>
>
> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <volume>
> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: qemu-img info
>
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
>
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
>
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) <volume>
> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Execution is successful.
> 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop
> it
> 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
>
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) <volume>
> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Try to stop the vm at first
> 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop
> it
> 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>
>
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Laszlo Hornyak <la...@gmail.com>.
Hi Andrija,
Could you also share management server logs?
On Sun, Mar 15, 2015 at 5:19 PM, Andrija Panic <an...@gmail.com>
wrote:
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
>
> can't find connection: KVM, for vm: r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
>
> - and BTW - absolutely NOTHING in the libvirt log !
>
>
> Here is the full log, with all the steps, on the host that doesnt host any
> other VMs (except one - so log is easy to read) - you can notice after 1st
> fail, agent copies new image/backing file, and tries again with new
> image...
>
> And BTW, same template is of course used for both System VR and VPC VRs...
>
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.44.253.13/var/lib/libvirt/SECONDARY
>
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> }
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid 'M2��N0�e���,�, need to create it
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Didn't find an existing storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> duplicate paths
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> <source>
> <host name='10.44.253.13'/>
> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> </source>
> <target>
> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> </target>
> </pool>
>
> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <volume>
> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: qemu-img info
>
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
>
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
>
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) <volume>
> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Execution is successful.
> 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop
> it
> 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
>
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
>
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
>
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) <volume>
> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
>
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Try to stop the vm at first
> 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop
> it
> 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>
>
>
>
>
> --
>
> Andrija Panić
>
--
EOF
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Hi,
Has the yum update touch any of your qemu/libvirt packages?
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> Sent: Sunday, 15 March, 2015 16:19:07
> Subject: [URGENT - HELP NEEDED]
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
>
> can't find connection: KVM, for vm: r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
>
> - and BTW - absolutely NOTHING in the libvirt log !
>
>
> Here is the full log, with all the steps, on the host that doesnt host any
> other VMs (except one - so log is easy to read) - you can notice after 1st
> fail, agent copies new image/backing file, and tries again with new image...
>
> And BTW, same template is of course used for both System VR and VPC VRs...
>
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.44.253.13/var/lib/libvirt/SECONDARY
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> }
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid 'M2��N0�e���,�, need to create it
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Didn't find an existing storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> duplicate paths
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> <source>
> <host name='10.44.253.13'/>
> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> </source>
> <target>
> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> </target>
> </pool>
>
> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <volume>
> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: qemu-img info
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) <volume>
> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Execution is successful.
> 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) <volume>
> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Try to stop the vm at first
> 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>
>
>
>
>
> --
>
> Andrija Panić
Re: [URGENT - HELP NEEDED]
Posted by Nux! <nu...@li.nux.ro>.
Hi,
Have you managed to get to the bottom of this?
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: users@cloudstack.apache.org, dev@cloudstack.apache.org
> Sent: Sunday, 15 March, 2015 16:19:07
> Subject: [URGENT - HELP NEEDED]
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors: