You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Kambiz Darabi <da...@m-creations.com> on 2014/03/21 23:36:44 UTC
Virtual Router doesn't start
Hello,
as this is my first post to the list, I would like to thank all
contributors for Cloudstack which I use since last fall without any
problems. I run 4.1.1 with KVM and advanced networking.
After a restart of the management server (stopping and starting the java
process), the virtual domain router doesn't start and
management-server.log shows a NullPointerException in
NetworkModelImpl.getIpInNetwork (cf. stack trace below).
By putting the server in debug mode and remote debugging, I found out
that the reason is a row in the table nics which has NULL in ip (cf. row
with id 30 in the result of the select statement below).
What can I do to quickly solve this problem? Any pointers or suggestions
are appreciated as the system is currently unusable.
Thank you for your help
Kambiz
management-server.log:
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking VirtualRouter to prepare for Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking Ovs to prepare for Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement] (Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated with any VPC
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking NiciraNvp to prepare for Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] (Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service Connectivity on network net1
2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] (Job-Executor-1:job-176) Service SecurityGroup is not supported in the network id=205
2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Lock is acquired for network id 202 as a part of network implement
2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Network id=202 is already implemented
2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Lock is released for network id 202 as a part of network implement
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking VirtualRouter to prepare for Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking Ovs to prepare for Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement] (Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated with any VPC
2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-176) Asking NiciraNvp to prepare for Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] (Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service Connectivity on network null
2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for VM[DomainRouter|r-19-VM]
2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-1:job-176) No need to recreate the volume: Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding disk to VM
2014-03-18 10:03:27,224 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0 gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99 eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1
2014-03-18 10:03:27,343 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR VM[DomainRouter|r-19-VM] start.
2014-03-18 10:03:27,415 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load balancing rules as a part of Virtual router start
2014-03-18 10:03:27,499 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a part of domR VM[DomainRouter|r-19-VM] start.
2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-176) Failed to start instance VM[DomainRouter|r-19-VM]
java.lang.NullPointerException
at com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:2435)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2855)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103)
table nics:
mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr |
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | 0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 | NULL | 202 | Static | Reserved | Start | ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 | 2014-03-17 22:21:38 | NULL | NULL | 0 | SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL | NULL |
| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | 0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 | NULL | 202 | Static | Reserved | Start | ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 | 2014-03-17 22:21:38 | NULL | NULL | 0 | ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL | NULL |
| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL | NULL | NULL | NULL | NULL | NULL | 202 | Static | Deallocating | Start | ControlNetworkGuru | c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 | NULL | NULL | 0 | DomainRouter | 2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL | NULL | NULL | NULL | NULL | NULL | 202 | Static | Deallocating | Start | ControlNetworkGuru | ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 | NULL | NULL | 0 | DomainRouter | 2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL | NULL | NULL | NULL | NULL | NULL | 202 | Static | Deallocating | Start | ControlNetworkGuru | ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 | NULL | NULL | 0 | DomainRouter | 2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL | NULL | NULL | NULL | NULL | NULL | 202 | Static | Allocated | Start | ControlNetworkGuru | e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 | NULL | NULL | 0 | DomainRouter | 2014-03-18 08:11:32 | NULL | NULL | NULL |
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
no I didn't expunge anything. The system was running with only the
console proxy and ssvm.
Is there something I can do to restore the functionality?
Shall I remove the static IPs from the stopped instances?
I have only 6 vm instances. I could even install from scratch if I knew
how to restore the state of the vms from their images. Is their a recipe
for this somewhere?
Thank you
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Kambiz, did you create a new static nat rule and expunged the vm? You
> should file a bug for:
>
> "Static rules not being cleared when user vm gts expunged"
>
> I’m not re how to resolve the oblem on the backend tugh as I’m not
> an expert when it comes to KVM agent code/ip rules programming. Edison,
> can yo help
>
> -Alena.
>
> On 3/2614, 3:19 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>it gets even stranger than it was. After manualg the iptables
>>commandyesterday, I had the folowing in agent.log
>>
>>http://pastie.org/8971483
>>
>>and I could start vm instances, but couldn't ping m.
>>
>>In the morning, changed the device IDs>
>>"nics":[
>> {"deviceId":2,
>> "defaultNic":true,
>> "ip":"10.193.17.190",
>> "type":"Public",
>> {"deviceId":1,
>> "ip":"10.124.99.1",
>> "type":"Guest",
>> {"deviceId":0,
>> "ip":"169.254.3.204",
>> "ype":"Control",
>>
>>but didn't observe any change.
>>
>>Now (~ 15 hours later), I tried to start the virtual router and again
>>find the NullPointrException in agent.lo(http://pastie.org/89757):
>>
>>2014-03-26 21::18,371 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>(Job-Executor-1:job-194) Failed to start VM[DomainRouter|r-19-V]
>>java.lang.NullPointerException
>> at
>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.ja> at
>>com.cloud.etwork.router.VirtualNeworkApplianceManagerImpl.finalizeNetwor
>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346
>>
>>So, we are back tosquare one. I ran this uery:
>>
>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>> n.state as nic_state, n.removed as nic_removed,
>> vm.state as vm_state, vm.removed as vm_removed
>>from user_ip_address uip
>> jon nics n on uip.vm_id = n.instance_id
>> join vm_instance vm on uip.vm_id = vm.id
>>where uip.id in (Select ip_address_id from firewall_rule fr
>>where
>>fr.networkd=205);
>>
>>+-------+----------+-----------------+--------------+-----------------
>>----+-----------+------------+
>>| vm_id | network_id |ip_address | nic_state | nic_removed
>> | vm_state | vm_removed |
>>+-------+------------+-------------------+--------------+-------------+-----------+-----------+
>>| 6 | 25 | 10.193.17.169 | Allocated | NULL
>> | Stopped | NULL |
>>| 10 | 205 | 103.17.136 | Allocatd | NULL
>> | Stoped | NULL |
>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>> | Stopped | NULL |
>>| 13 | 205 | 10.193.17.141 | Deallocating | 2014-03-26
>>07:03:27 | Expunging | NLL |
>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>> | Stopped | NULL |
>>| 16 | 205 | 10.193.17.13 | Allocated |ULL
>> | Stopped |ULL |
>>+-------+----------+-------------------+--------------+-----------------
>>----+-----------+-----------I know that I would hav to run this, next:
>>
>>update user_ip_address set one_to_one_nat=0, vm_id=null where vm_id=13;
>>
>>
>>But what are my options to get to a working system again? I'm struggling
>>with this issue since days.
>>
>>Thank you
>>
>>
>>Kambiz
>>
>>Kambiz Darabi <da...@m-creations.com> wrote:
>>>
>>> Yes, I know, but the messages do not stop, the router remains in state
>>> Starting for some time and is then shut down, because it couldn't be
>>> contacted.
>>>
>>> The problem is not yet completely solved. To make sshd listen all an
>>> interfaces, I did the following manually in the router VM:
>>>
>>> echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
>>> /etc/init.d/ssh restart
>>> iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 3922 -j
>>>ACCEPT
>>>
>>> After this, the router becomes 'Running' in the UI, agent.log says
>>>
>>> Ping command port succeeded for vm r-19-VM
>>> Executing:
>>>/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>get_template_version.sh 169.254.3.203
>>> Execution is successful.
>>>
>>> Maybe I set the wrong deviceId to 1 and should have taken the other nic
>>> with the deviceId 0. I will try that tomorrow and report back to you.
>>>
>>> Thank you so much for your help and patience.
>>>
>>>
>>>
>>> Kambiz
>>>
>>>
>>> Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> Seeing these messages for some time is fine as some delay in startup
>>>> process is expected, and eventually the command will succeed.
>>>>
>>>> telnet 169.254.3.155 3922
>>>> Trying 169.254.3.155…’
>>>>
>>>> So it works like this:
>>>>
>>>> 1) Start router is executed. Router is booting up
>>>> 2) Agent tries to ssh to the VR to ensure that the router is up
>>>> 3) Once its up, agent reports to the MS that startup was successful.
>>>>
>>>> As long as VR startup is reported as success to the MS, everything is
>>>>good
>>>> and you shouldn’t worry about those messages.
>>>>
>>>> -Alena.
>>>>
>>>>
>>>>
>>>> On 3/25/14, 3:22 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>I executed
>>>>>
>>>>>> update nics set device_id = 1 where id = 29;
>>>>>
>>>>>After restarting the router, the interfaces file now looks like this:
>>>>>
>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>auto lo eth0 eth1 eth2
>>>>>iface lo inet loopback
>>>>>
>>>>>iface eth0 inet static
>>>>> address 169.254.3.155
>>>>> netmask 255.255.0.0
>>>>>iface eth1 inet static
>>>>> address 10.124.99.1
>>>>> netmask 255.255.255.0
>>>>>iface eth2 inet static
>>>>> address 10.193.17.190
>>>>> netmask 255.255.255.0
>>>>>
>>>>>ifconfig shows this:
>>>>>
>>>>>root@r-19-VM:~# ifconfig
>>>>>eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
>>>>> inet addr:169.254.3.155 Bcast:169.254.255.255
>>>>>Mask:255.255.0.0
>>>>> inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
>>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>>> RX packets:61 errors:0 dropped:0 overruns:0 frame:0
>>>>> TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
>>>>> collisions:0 txqueuelen:1000
>>>>> RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
>>>>>
>>>>>eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
>>>>> inet addr:10.124.99.1 Bcast:10.124.99.255
>>>>>Mask:255.255.255.0
>>>>> inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
>>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>>> RX packets:12 errors:0 dropped:0 overruns:0 frame:0
>>>>> TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
>>>>> collisions:0 txqueuelen:1000
>>>>> RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
>>>>>
>>>>>eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
>>>>> inet addr:10.193.17.190 Bcast:10.193.17.255
>>>>>Mask:255.255.255.0
>>>>> inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
>>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>>> RX packets:11 errors:0 dropped:0 overruns:0 frame:0
>>>>> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>>>>> collisions:0 txqueuelen:1000
>>>>> RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>>>>>
>>>>>From inside the VM, I can ping the gateways of the public, guest and
>>>>>control network:
>>>>>
>>>>>root@r-19-VM:~# route -n
>>>>>Kernel IP routing table
>>>>>Destination Gateway Genmask Flags Metric Ref Use
>>>>>Iface
>>>>>10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>>>eth1
>>>>>10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>>>eth2
>>>>>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
>>>>>eth0
>>>>>0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0
>>>>>eth2
>>>>>
>>>>>root@r-19-VM:~# ping 10.193.17.1
>>>>>PING 10.193.17.1 (10.193.17.1): 56 data bytes
>>>>>64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
>>>>>64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
>>>>>
>>>>>root@r-19-VM:~# ping 10.124.99.1
>>>>>PING 10.124.99.1 (10.124.99.1): 56 data bytes
>>>>>64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
>>>>>
>>>>>root@r-19-VM:/etc/init.d# ping 169.254.0.1
>>>>>PING 169.254.0.1 (169.254.0.1): 56 data bytes
>>>>>64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
>>>>>
>>>>>And from outside, I can ping the different IPs of the router.
>>>>>
>>>>>But what is strange, is that in agent.log, I still find
>>>>>
>>>>>Ping command port, 169.254.3.155:3922
>>>>>Trying to connect to 169.254.3.155
>>>>>Could not connect to 169.254.3.155
>>>>>
>>>>>And when I check on the router, the ssh daemon only listens on the
>>>>>guest
>>>>>network interface:
>>>>>
>>>>># netstat -na | grep 3922
>>>>>tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
>>>>>
>>>>>So, the connection attempt to 169.254.3.155:3922 fails:
>>>>>
>>>>>telnet 169.254.3.155 3922
>>>>>Trying 169.254.3.155...
>>>>>
>>>>>Is that the normal situation?
>>>>>
>>>>>> Also after which point you started experiencing all this problems?
>>>>>>Did
>>>>>> you upgrade to new CS version? Or does it fail for any specific
>>>>>>network?
>>>>>
>>>>>No, I didn't upgrade to a new CS, I just stopped and started the
>>>>>management-server.
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> Kambiz, did you check the device id in nics table for the vm? If it
>>>>>>has
>>>>>>2
>>>>>> 0s, change one of them to the correct value and restart. If the
>>>>>>testing
>>>>>> completes fine, we have a proof that its related to the device id mix
>>>>>>up.
>>>>>> If not, there is gotta be something else, most likely misconfigured
>>>>>>on
>>>>>>KVM
>>>>>> stuff.
>>>>>>
>>>>>> Also after which point you started experiencing all this problems?
>>>>>>Did
>>>>>> you upgrade to new CS version? Or does it fail for any specific
>>>>>>network?
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>I updated nics.gateway for that network, but the VM still shows the
>>>>>>>same
>>>>>>>behaviour.
>>>>>>>
>>>>>>>If one compares interfaces:
>>>>>>>
>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>iface lo inet loopback
>>>>>>>
>>>>>>>iface eth0 inet static
>>>>>>> address 169.254.1.242
>>>>>>> netmask 255.255.0.0
>>>>>>>iface eth1 inet static
>>>>>>> address 10.193.17.1
>>>>>>> netmask
>>>>>>>iface eth2 inet static
>>>>>>> address 10.193.17.190
>>>>>>> netmask 255.255.255.0
>>>>>>>
>>>>>>>and the nics entry in the management-server.log (cf. below), one can
>>>>>>>see
>>>>>>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2
>>>>>>>is
>>>>>>>correctly set up with IP 10.193.17.190.
>>>>>>>
>>>>>>>The remaining nic is eth1 which corresponds to the first nic with
>>>>>>>deviceId 0 and according to the nics entry should have IP
>>>>>>>10.124.99.1,
>>>>>>>but there is no iface entry with that IP, but eth1 has the gateway
>>>>>>>address of the public nic as its IP address.
>>>>>>>
>>>>>>>Could the problem have something to do with the duplicate deviceId 0?
>>>>>>>
>>>>>>>Thanks
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>{ "nics":
>>>>>>> [{"deviceId":2,
>>>>>>> "networkRateMbps":200,
>>>>>>> "defaultNic":true,
>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>> "ip":"10.193.17.190",
>>>>>>> "netmask":"255.255.255.0",
>>>>>>> "gateway":"10.193.17.1",
>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>> "dns1":"10.193.17.1",
>>>>>>> "broadcastType":"Vlan",
>>>>>>> "type":"Public",
>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>> "name":"cloudbr0"},
>>>>>>> {"deviceId":0,
>>>>>>> "networkRateMbps":200,
>>>>>>> "defaultNic":false,
>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>> "ip":"10.124.99.1",
>>>>>>> "netmask":"255.255.255.0",
>>>>>>> "gateway":"10.124.99.1",
>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>> "dns1":"10.193.17.1",
>>>>>>> "broadcastType":"Vlan",
>>>>>>> "type":"Guest",
>>>>>>> "broadcastUri":"vlan://3925",
>>>>>>> "isolationUri":"vlan://3925",
>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>> "name":"cloudbr1"},
>>>>>>> {"deviceId":0,
>>>>>>> "networkRateMbps":-1,
>>>>>>> "defaultNic":false,
>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>> "ip":"169.254.1.242",
>>>>>>> "netmask":"255.255.0.0",
>>>>>>> "gateway":"169.254.0.1",
>>>>>>> "mac":"0e:00:a9:fe:01:f2",
>>>>>>> "broadcastType":"LinkLocal",
>>>>>>> "type":"Control",
>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>> ]
>>>>>>>}
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> So the gateway wasn’t set for the nic only.
>>>>>>>>
>>>>>>>> Kambiz, just to quickly test it, can you set missing gateway for
>>>>>>>>the
>>>>>>>>nic,
>>>>>>>> and stop/start the VR? And see if the start is completed normally,
>>>>>>>>just
>>>>>>>>to
>>>>>>>> find out if the missing gateway was the reason of the communication
>>>>>>>>failure
>>>>>>>>
>>>>>>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>select id, name, traffic_type, broadcast_domain_type, cidr,
>>>>>>>>>gateway,
>>>>>>>>>mode, state, removed from networks where id = 205;
>>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>>---
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>------------+------+-----------+---------+
>>>>>>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>>>>>>|
>>>>>>>>>gateway | mode | state | removed |
>>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>>---
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>------------+------+-----------+---------+
>>>>>>>>>| 205 | default | Guest | Vlan |
>>>>>>>>>10.124.99.0/24
>>>>>>>>>|
>>>>>>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>>---
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>------------+------+-----------+---------+
>>>>>>>>>
>>>>>>>>>Cheers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> No, it doesn’t seem right to me having 2 nics with device id 0.
>>>>>>>>>>But
>>>>>>>>>>looks
>>>>>>>>>> like they’ve got programmed to correct devices on the backend per
>>>>>>>>>>your
>>>>>>>>>> prev email?
>>>>>>>>>>
>>>>>>>>>> iface eth0 inet static
>>>>>>>>>> address 169.254.1.59
>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>
>>>>>>>>>> iface eth1 inet static
>>>>>>>>>> address 10.193.17.1
>>>>>>>>>> Netmask
>>>>>>>>>>
>>>>>>>>>> iface eth2 inet static
>>>>>>>>>> address 10.193.17.190
>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I can see that only one parameter is missing from the start
>>>>>>>>>>command,
>>>>>>>>>>the
>>>>>>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>>>>>>> From the command/DB, I see that the gateway is missing in the
>>>>>>>>>>nics
>>>>>>>>>>table
>>>>>>>>>> for the network 205? Can you check gateway information in the
>>>>>>>>>>networks
>>>>>>>>>> table for the id=205
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi,
>>>>>>>>>>>
>>>>>>>>>>>select
>>>>>>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_
>>>>>>>>>>>nam
>>>>>>>>>>>e
>>>>>>>>>>>from nics where instance_id=19;
>>>>>>>>>>>
>>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>>---
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>>>>>>removed
>>>>>>>>>>>| network_id | reserver_name |
>>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>>---
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated |
>>>>>>>>>>>NULL
>>>>>>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>>>>>>| 30 | NULL | NULL | NULL | Allocated |
>>>>>>>>>>>NULL
>>>>>>>>>>>| 202 | ControlNetworkGuru |
>>>>>>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated |
>>>>>>>>>>>NULL
>>>>>>>>>>>| 200 | PublicNetworkGuru |
>>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>>---
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>>
>>>>>>>>>>>and this is the nics element from the StartCmd. Is it normal to
>>>>>>>>>>>have
>>>>>>>>>>>two nics with deviceId 0?
>>>>>>>>>>>
>>>>>>>>>>>"nics":[
>>>>>>>>>>> {"deviceId":2,
>>>>>>>>>>> "networkRateMbps":200,
>>>>>>>>>>> "defaultNic":true,
>>>>>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>>>>>> "ip":"10.193.17.190",
>>>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>>>> "gateway":"10.193.17.1",
>>>>>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>>>> "type":"Public",
>>>>>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>>>> "name":"cloudbr0"},
>>>>>>>>>>> {"deviceId":0,
>>>>>>>>>>> "networkRateMbps":200,
>>>>>>>>>>> "defaultNic":false,
>>>>>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>>>>>> "ip":"10.124.99.1",
>>>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>>>> "type":"Guest",
>>>>>>>>>>> "broadcastUri":"vlan://3949",
>>>>>>>>>>> "isolationUri":"vlan://3949",
>>>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>>>> "name":"cloudbr1"},
>>>>>>>>>>> {"deviceId":0,
>>>>>>>>>>> "networkRateMbps":-1,
>>>>>>>>>>> "defaultNic":false,
>>>>>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>>>>>> "ip":"169.254.1.59",
>>>>>>>>>>> "netmask":"255.255.0.0",
>>>>>>>>>>> "gateway":"169.254.0.1",
>>>>>>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>>>>>>> "broadcastType":"LinkLocal",
>>>>>>>>>>> "type":"Control",
>>>>>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>>>>>>]
>>>>>>>>>>>
>>>>>>>>>>>Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Kambiz, the debug statements below are for the case when eth1
>>>>>>>>>>>>is a
>>>>>>>>>>>>control
>>>>>>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>>>>>>command,
>>>>>>>>>>>> eth1 is not control, its either public or guest
>>>>>>>>>>>>
>>>>>>>>>>>> eth0: - control
>>>>>>>>>>>>
>>>>>>>>>>>> iface eth0 inet static
>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>
>>>>>>>>>>>> eth1:
>>>>>>>>>>>>
>>>>>>>>>>>> iface eth1 inet static
>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>> Netmask
>>>>>>>>>>>>
>>>>>>>>>>>> So you need to execute the mysql statements for the traffic
>>>>>>>>>>>>type
>>>>>>>>>>>>of
>>>>>>>>>>>>VR
>>>>>>>>>>>>nic
>>>>>>>>>>>> eth1
>>>>>>>>>>>>
>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>>>>>>><Al...@citrix.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>>>>>>
>>>>>>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>>>>>>traffic_type=‘Control’;
>>>>>>>>>>>>>
>>>>>>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid>
>>>>>>>>>>>>>and
>>>>>>>>>>>>>network_id=<control network id>
>>>>>>>>>>>>>
>>>>>>>>>>>>>-Alena.
>>>>>>>>>>>>>
>>>>>>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I looked up the startup command of the old router instance
>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>worked
>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomr
>>>>>>>>>>>>>>pre
>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>h
>>>>>>>>>>>>>>-l
>>>>>>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.2
>>>>>>>>>>>>>>55.
>>>>>>>>>>>>>>25
>>>>>>>>>>>>>>5.
>>>>>>>>>>>>>>0%
>>>>>>>>>>>>>>ga
>>>>>>>>>>>>>>t
>>>>>>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%dom
>>>>>>>>>>>>>>ain
>>>>>>>>>>>>>>=c
>>>>>>>>>>>>>>s6
>>>>>>>>>>>>>>cl
>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>d
>>>>>>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=2
>>>>>>>>>>>>>>55.
>>>>>>>>>>>>>>25
>>>>>>>>>>>>>>5.
>>>>>>>>>>>>>>0.
>>>>>>>>>>>>>>0%
>>>>>>>>>>>>>>t
>>>>>>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>eth1mask.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>>>>>>eliminated
>>>>>>>>>>>>>>> static nat rule programming for non-existing vm. Missing
>>>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>>>on
>>>>>>>>>>>>>>>eth1
>>>>>>>>>>>>>>> doesn’t seem related to the problem (although we have to
>>>>>>>>>>>>>>>figure
>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>why
>>>>>>>>>>>>>>> its missing), as the connection that fails, happening to
>>>>>>>>>>>>>>>link
>>>>>>>>>>>>>>>local
>>>>>>>>>>>>>>>169.x
>>>>>>>>>>>>>>> eth0 interface.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Edison, can you please tell us how to debug link local
>>>>>>>>>>>>>>>connection
>>>>>>>>>>>>>>>failure,
>>>>>>>>>>>>>>> on KVM agent?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>> Alena.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more,
>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>When I start the virtual router, it comes up, but in
>>>>>>>>>>>>>>>>agent.log,
>>>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>>>>>>...
>>>>>>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>>>>>> netmask
>>>>>>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>>>>>>agent.log
>>>>>>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundo
>>>>>>>>>>>>>>>>mrp
>>>>>>>>>>>>>>>>re
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>h
>>>>>>>>>>>>>>>>-l
>>>>>>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk
>>>>>>>>>>>>>>>>-p
>>>>>>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=25
>>>>>>>>>>>>>>>>5.2
>>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>>>%
>>>>>>>>>>>>>>>>ga
>>>>>>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%
>>>>>>>>>>>>>>>>dom
>>>>>>>>>>>>>>>>ai
>>>>>>>>>>>>>>>>n=
>>>>>>>>>>>>>>>>cs
>>>>>>>>>>>>>>>>6c
>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mas
>>>>>>>>>>>>>>>>k=2
>>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>>>.
>>>>>>>>>>>>>>>>0%
>>>>>>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>>>>>>culprit.
>>>>>>>>>>>>>>>>>If
>>>>>>>>>>>>>>>>>vm
>>>>>>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to
>>>>>>>>>>>>>>>>>it
>>>>>>>>>>>>>>>>>from
>>>>>>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0,
>>>>>>>>>>>>>>>>>instance_id=null
>>>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>>>>>>following
>>>>>>>>>>>>>>>>>>query
>>>>>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules
>>>>>>>>>>>>>>>>>>fr
>>>>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated |
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network
>>>>>>>>>>>>>>>>>>>(205,
>>>>>>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is
>>>>>>>>>>>>>>>>>>>non-removed
>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static
>>>>>>>>>>>>>>>>>>>nat
>>>>>>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>>>>>>> referring to vm that is removed already. If you find
>>>>>>>>>>>>>>>>>>>any,
>>>>>>>>>>>>>>>>>>>let
>>>>>>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not
>>>>>>>>>>>>>>>>>>>>null,
>>>>>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router
>>>>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>>>>>>was used by my instances. I deleted the router using
>>>>>>>>>>>>>>>>>>>>the UI
>>>>>>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar
>>>>>>>>>>>>>>>>>>>>problem
>>>>>>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>>>>>>that the deleted router would be recreated again (and
>>>>>>>>>>>>>>>>>>>>this
>>>>>>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because
>>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>VPC
>>>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to
>>>>>>>>>>>>>>>>>>>>>revert
>>>>>>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id
>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It will give you the list of networks refs that
>>>>>>>>>>>>>>>>>>>>>somehow
>>>>>>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to
>>>>>>>>>>>>>>>>>>>>>>thank
>>>>>>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote
>>>>>>>>>>>>>>>>>>>>>>debugging, I
>>>>>>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is
>>>>>>>>>>>>>>>>>>>>>>not
>>>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a
>>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress
>>>>>>>>>>>>>>>>>>>>>>rule(s)
>>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(Network
>>>>>>>>>>>>>>>>>>>>>>Mo
>>>>>>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>>>>>>>>>>23
>>>>>>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:7
>>>>>>>>>>>>>>>>>>>>>>67
>>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtu
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachi
>>>>>>>>>>>>>>>>>>>>>>ne
>>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.ja
>>>>>>>>>>>>>>>>>>>>>>va
>>>>>>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(
>>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElem
>>>>>>>>>>>>>>>>>>>>>>en
>>>>>>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Net
>>>>>>>>>>>>>>>>>>>>>>wo
>>>>>>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(Network
>>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRoute
>>>>>>>>>>>>>>>>>>>>>>rC
>>>>>>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>>| id | uuid |
>>>>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway |
>>>>>>>>>>>>>>>>>>>>>>ip_type |
>>>>>>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>>>>> | removed | ip6_gateway |
>>>>>>>>>>>>>>>>>>>>>>ip6_cidr |
>>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |
>>>>>>>>>>>>>>>>>>>>>>1 |
>>>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |
>>>>>>>>>>>>>>>>>>>>>>2 |
>>>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |
>>>>>>>>>>>>>>>>>>>>>>4
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |
>>>>>>>>>>>>>>>>>>>>>>7
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |
>>>>>>>>>>>>>>>>>>>>>>18
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |
>>>>>>>>>>>>>>>>>>>>>>19
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Kambiz, did you create a new static nat rule and expunged the vm? You
should file a bug for:
"Static rules not being cleared when user vm gts expunged"
I’m not re how to resolve the oblem on the backend tugh as I’m not
an expert when it comes to KVM agent code/ip rules programming. Edison,
can yo help
-Alena.
On 3/2614, 3:19 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>it gets even stranger than it was. After manualg the iptables
>commandyesterday, I had the folowing in agent.log
>
>http://pastie.org/8971483
>
>and I could start vm instances, but couldn't ping m.
>
>In the morning, changed the device IDs>
>"nics":[
> {"deviceId":2,
> "defaultNic":true,
> "ip":"10.193.17.190",
> "type":"Public",
> {"deviceId":1,
> "ip":"10.124.99.1",
> "type":"Guest",
> {"deviceId":0,
> "ip":"169.254.3.204",
> "ype":"Control",
>
>but didn't observe any change.
>
>Now (~ 15 hours later), I tried to start the virtual router and again
>find the NullPointrException in agent.lo(http://pastie.org/89757):
>
>2014-03-26 21::18,371 ERROR [cloud.vm.VirtualMachineManagerImpl]
>(Job-Executor-1:job-194) Failed to start VM[DomainRouter|r-19-V]
>java.lang.NullPointerException
> at
>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.ja> at
>com.cloud.etwork.router.VirtualNeworkApplianceManagerImpl.finalizeNetwor
>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346
>
>So, we are back tosquare one. I ran this uery:
>
>select uip.vm_id, uip.network_id, uip.public_ip_address,
> n.state as nic_state, n.removed as nic_removed,
> vm.state as vm_state, vm.removed as vm_removed
>from user_ip_address uip
> jon nics n on uip.vm_id = n.instance_id
> join vm_instance vm on uip.vm_id = vm.id
>where uip.id in (Select ip_address_id from firewall_rule fr
>where
>fr.networkd=205);
>
>+-------+----------+-----------------+--------------+-----------------
>----+-----------+------------+
>| vm_id | network_id |ip_address | nic_state | nic_removed
> | vm_state | vm_removed |
>+-------+------------+-------------------+--------------+-------------+-----------+-----------+
>| 6 | 25 | 10.193.17.169 | Allocated | NULL
> | Stopped | NULL |
>| 10 | 205 | 103.17.136 | Allocatd | NULL
> | Stoped | NULL |
>| 12 | 205 | 10.193.17.140 | Allocated | NULL
> | Stopped | NULL |
>| 13 | 205 | 10.193.17.141 | Deallocating | 2014-03-26
>07:03:27 | Expunging | NLL |
>| 14 | 205 | 10.193.17.142 | Allocated | NULL
> | Stopped | NULL |
>| 16 | 205 | 10.193.17.13 | Allocated |ULL
> | Stopped |ULL |
>+-------+----------+-------------------+--------------+-----------------
>----+-----------+-----------I know that I would hav to run this, next:
>
>update user_ip_address set one_to_one_nat=0, vm_id=null where vm_id=13;
>
>
>But what are my options to get to a working system again? I'm struggling
>with this issue since days.
>
>Thank you
>
>
>Kambiz
>
>Kambiz Darabi <da...@m-creations.com> wrote:
>>
>> Yes, I know, but the messages do not stop, the router remains in state
>> Starting for some time and is then shut down, because it couldn't be
>> contacted.
>>
>> The problem is not yet completely solved. To make sshd listen all an
>> interfaces, I did the following manually in the router VM:
>>
>> echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
>> /etc/init.d/ssh restart
>> iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 3922 -j
>>ACCEPT
>>
>> After this, the router becomes 'Running' in the UI, agent.log says
>>
>> Ping command port succeeded for vm r-19-VM
>> Executing:
>>/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>get_template_version.sh 169.254.3.203
>> Execution is successful.
>>
>> Maybe I set the wrong deviceId to 1 and should have taken the other nic
>> with the deviceId 0. I will try that tomorrow and report back to you.
>>
>> Thank you so much for your help and patience.
>>
>>
>>
>> Kambiz
>>
>>
>> Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> Seeing these messages for some time is fine as some delay in startup
>>> process is expected, and eventually the command will succeed.
>>>
>>> telnet 169.254.3.155 3922
>>> Trying 169.254.3.155…’
>>>
>>> So it works like this:
>>>
>>> 1) Start router is executed. Router is booting up
>>> 2) Agent tries to ssh to the VR to ensure that the router is up
>>> 3) Once its up, agent reports to the MS that startup was successful.
>>>
>>> As long as VR startup is reported as success to the MS, everything is
>>>good
>>> and you shouldn’t worry about those messages.
>>>
>>> -Alena.
>>>
>>>
>>>
>>> On 3/25/14, 3:22 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>I executed
>>>>
>>>>> update nics set device_id = 1 where id = 29;
>>>>
>>>>After restarting the router, the interfaces file now looks like this:
>>>>
>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>auto lo eth0 eth1 eth2
>>>>iface lo inet loopback
>>>>
>>>>iface eth0 inet static
>>>> address 169.254.3.155
>>>> netmask 255.255.0.0
>>>>iface eth1 inet static
>>>> address 10.124.99.1
>>>> netmask 255.255.255.0
>>>>iface eth2 inet static
>>>> address 10.193.17.190
>>>> netmask 255.255.255.0
>>>>
>>>>ifconfig shows this:
>>>>
>>>>root@r-19-VM:~# ifconfig
>>>>eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
>>>> inet addr:169.254.3.155 Bcast:169.254.255.255
>>>>Mask:255.255.0.0
>>>> inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>> RX packets:61 errors:0 dropped:0 overruns:0 frame:0
>>>> TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
>>>> collisions:0 txqueuelen:1000
>>>> RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
>>>>
>>>>eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
>>>> inet addr:10.124.99.1 Bcast:10.124.99.255
>>>>Mask:255.255.255.0
>>>> inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>> RX packets:12 errors:0 dropped:0 overruns:0 frame:0
>>>> TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
>>>> collisions:0 txqueuelen:1000
>>>> RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
>>>>
>>>>eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
>>>> inet addr:10.193.17.190 Bcast:10.193.17.255
>>>>Mask:255.255.255.0
>>>> inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>>> RX packets:11 errors:0 dropped:0 overruns:0 frame:0
>>>> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>>>> collisions:0 txqueuelen:1000
>>>> RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>>>>
>>>>From inside the VM, I can ping the gateways of the public, guest and
>>>>control network:
>>>>
>>>>root@r-19-VM:~# route -n
>>>>Kernel IP routing table
>>>>Destination Gateway Genmask Flags Metric Ref Use
>>>>Iface
>>>>10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>>eth1
>>>>10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>>eth2
>>>>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
>>>>eth0
>>>>0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0
>>>>eth2
>>>>
>>>>root@r-19-VM:~# ping 10.193.17.1
>>>>PING 10.193.17.1 (10.193.17.1): 56 data bytes
>>>>64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
>>>>64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
>>>>
>>>>root@r-19-VM:~# ping 10.124.99.1
>>>>PING 10.124.99.1 (10.124.99.1): 56 data bytes
>>>>64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
>>>>
>>>>root@r-19-VM:/etc/init.d# ping 169.254.0.1
>>>>PING 169.254.0.1 (169.254.0.1): 56 data bytes
>>>>64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
>>>>
>>>>And from outside, I can ping the different IPs of the router.
>>>>
>>>>But what is strange, is that in agent.log, I still find
>>>>
>>>>Ping command port, 169.254.3.155:3922
>>>>Trying to connect to 169.254.3.155
>>>>Could not connect to 169.254.3.155
>>>>
>>>>And when I check on the router, the ssh daemon only listens on the
>>>>guest
>>>>network interface:
>>>>
>>>># netstat -na | grep 3922
>>>>tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
>>>>
>>>>So, the connection attempt to 169.254.3.155:3922 fails:
>>>>
>>>>telnet 169.254.3.155 3922
>>>>Trying 169.254.3.155...
>>>>
>>>>Is that the normal situation?
>>>>
>>>>> Also after which point you started experiencing all this problems?
>>>>>Did
>>>>> you upgrade to new CS version? Or does it fail for any specific
>>>>>network?
>>>>
>>>>No, I didn't upgrade to a new CS, I just stopped and started the
>>>>management-server.
>>>>
>>>>Thanks
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> Kambiz, did you check the device id in nics table for the vm? If it
>>>>>has
>>>>>2
>>>>> 0s, change one of them to the correct value and restart. If the
>>>>>testing
>>>>> completes fine, we have a proof that its related to the device id mix
>>>>>up.
>>>>> If not, there is gotta be something else, most likely misconfigured
>>>>>on
>>>>>KVM
>>>>> stuff.
>>>>>
>>>>> Also after which point you started experiencing all this problems?
>>>>>Did
>>>>> you upgrade to new CS version? Or does it fail for any specific
>>>>>network?
>>>>>
>>>>> -Alena.
>>>>>
>>>>> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>I updated nics.gateway for that network, but the VM still shows the
>>>>>>same
>>>>>>behaviour.
>>>>>>
>>>>>>If one compares interfaces:
>>>>>>
>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>auto lo eth0 eth1 eth2
>>>>>>iface lo inet loopback
>>>>>>
>>>>>>iface eth0 inet static
>>>>>> address 169.254.1.242
>>>>>> netmask 255.255.0.0
>>>>>>iface eth1 inet static
>>>>>> address 10.193.17.1
>>>>>> netmask
>>>>>>iface eth2 inet static
>>>>>> address 10.193.17.190
>>>>>> netmask 255.255.255.0
>>>>>>
>>>>>>and the nics entry in the management-server.log (cf. below), one can
>>>>>>see
>>>>>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2
>>>>>>is
>>>>>>correctly set up with IP 10.193.17.190.
>>>>>>
>>>>>>The remaining nic is eth1 which corresponds to the first nic with
>>>>>>deviceId 0 and according to the nics entry should have IP
>>>>>>10.124.99.1,
>>>>>>but there is no iface entry with that IP, but eth1 has the gateway
>>>>>>address of the public nic as its IP address.
>>>>>>
>>>>>>Could the problem have something to do with the duplicate deviceId 0?
>>>>>>
>>>>>>Thanks
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>{ "nics":
>>>>>> [{"deviceId":2,
>>>>>> "networkRateMbps":200,
>>>>>> "defaultNic":true,
>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>> "ip":"10.193.17.190",
>>>>>> "netmask":"255.255.255.0",
>>>>>> "gateway":"10.193.17.1",
>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>> "dns1":"10.193.17.1",
>>>>>> "broadcastType":"Vlan",
>>>>>> "type":"Public",
>>>>>> "broadcastUri":"vlan://untagged",
>>>>>> "isolationUri":"vlan://untagged",
>>>>>> "isSecurityGroupEnabled":false,
>>>>>> "name":"cloudbr0"},
>>>>>> {"deviceId":0,
>>>>>> "networkRateMbps":200,
>>>>>> "defaultNic":false,
>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>> "ip":"10.124.99.1",
>>>>>> "netmask":"255.255.255.0",
>>>>>> "gateway":"10.124.99.1",
>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>> "dns1":"10.193.17.1",
>>>>>> "broadcastType":"Vlan",
>>>>>> "type":"Guest",
>>>>>> "broadcastUri":"vlan://3925",
>>>>>> "isolationUri":"vlan://3925",
>>>>>> "isSecurityGroupEnabled":false,
>>>>>> "name":"cloudbr1"},
>>>>>> {"deviceId":0,
>>>>>> "networkRateMbps":-1,
>>>>>> "defaultNic":false,
>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>> "ip":"169.254.1.242",
>>>>>> "netmask":"255.255.0.0",
>>>>>> "gateway":"169.254.0.1",
>>>>>> "mac":"0e:00:a9:fe:01:f2",
>>>>>> "broadcastType":"LinkLocal",
>>>>>> "type":"Control",
>>>>>> "isSecurityGroupEnabled":false}
>>>>>> ]
>>>>>>}
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> So the gateway wasn’t set for the nic only.
>>>>>>>
>>>>>>> Kambiz, just to quickly test it, can you set missing gateway for
>>>>>>>the
>>>>>>>nic,
>>>>>>> and stop/start the VR? And see if the start is completed normally,
>>>>>>>just
>>>>>>>to
>>>>>>> find out if the missing gateway was the reason of the communication
>>>>>>>failure
>>>>>>>
>>>>>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>select id, name, traffic_type, broadcast_domain_type, cidr,
>>>>>>>>gateway,
>>>>>>>>mode, state, removed from networks where id = 205;
>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>---
>>>>>>>>--
>>>>>>>>+-
>>>>>>>>------------+------+-----------+---------+
>>>>>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>>>>>|
>>>>>>>>gateway | mode | state | removed |
>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>---
>>>>>>>>--
>>>>>>>>+-
>>>>>>>>------------+------+-----------+---------+
>>>>>>>>| 205 | default | Guest | Vlan |
>>>>>>>>10.124.99.0/24
>>>>>>>>|
>>>>>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>>>>>+-----+---------+--------------+-----------------------+-----------
>>>>>>>>---
>>>>>>>>--
>>>>>>>>+-
>>>>>>>>------------+------+-----------+---------+
>>>>>>>>
>>>>>>>>Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> No, it doesn’t seem right to me having 2 nics with device id 0.
>>>>>>>>>But
>>>>>>>>>looks
>>>>>>>>> like they’ve got programmed to correct devices on the backend per
>>>>>>>>>your
>>>>>>>>> prev email?
>>>>>>>>>
>>>>>>>>> iface eth0 inet static
>>>>>>>>> address 169.254.1.59
>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>
>>>>>>>>> iface eth1 inet static
>>>>>>>>> address 10.193.17.1
>>>>>>>>> Netmask
>>>>>>>>>
>>>>>>>>> iface eth2 inet static
>>>>>>>>> address 10.193.17.190
>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can see that only one parameter is missing from the start
>>>>>>>>>command,
>>>>>>>>>the
>>>>>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>>>>>> From the command/DB, I see that the gateway is missing in the
>>>>>>>>>nics
>>>>>>>>>table
>>>>>>>>> for the network 205? Can you check gateway information in the
>>>>>>>>>networks
>>>>>>>>> table for the id=205
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>select
>>>>>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_
>>>>>>>>>>nam
>>>>>>>>>>e
>>>>>>>>>>from nics where instance_id=19;
>>>>>>>>>>
>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>---
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>>>>>removed
>>>>>>>>>>| network_id | reserver_name |
>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>---
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated |
>>>>>>>>>>NULL
>>>>>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>>>>>| 30 | NULL | NULL | NULL | Allocated |
>>>>>>>>>>NULL
>>>>>>>>>>| 202 | ControlNetworkGuru |
>>>>>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated |
>>>>>>>>>>NULL
>>>>>>>>>>| 200 | PublicNetworkGuru |
>>>>>>>>>>+----+---------------+---------------+-------------+-----------+-
>>>>>>>>>>---
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>------------+--------------------------+
>>>>>>>>>>
>>>>>>>>>>and this is the nics element from the StartCmd. Is it normal to
>>>>>>>>>>have
>>>>>>>>>>two nics with deviceId 0?
>>>>>>>>>>
>>>>>>>>>>"nics":[
>>>>>>>>>> {"deviceId":2,
>>>>>>>>>> "networkRateMbps":200,
>>>>>>>>>> "defaultNic":true,
>>>>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>>>>> "ip":"10.193.17.190",
>>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>>> "gateway":"10.193.17.1",
>>>>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>>> "type":"Public",
>>>>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>>> "name":"cloudbr0"},
>>>>>>>>>> {"deviceId":0,
>>>>>>>>>> "networkRateMbps":200,
>>>>>>>>>> "defaultNic":false,
>>>>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>>>>> "ip":"10.124.99.1",
>>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>>> "type":"Guest",
>>>>>>>>>> "broadcastUri":"vlan://3949",
>>>>>>>>>> "isolationUri":"vlan://3949",
>>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>>> "name":"cloudbr1"},
>>>>>>>>>> {"deviceId":0,
>>>>>>>>>> "networkRateMbps":-1,
>>>>>>>>>> "defaultNic":false,
>>>>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>>>>> "ip":"169.254.1.59",
>>>>>>>>>> "netmask":"255.255.0.0",
>>>>>>>>>> "gateway":"169.254.0.1",
>>>>>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>>>>>> "broadcastType":"LinkLocal",
>>>>>>>>>> "type":"Control",
>>>>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>>>>>]
>>>>>>>>>>
>>>>>>>>>>Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Kambiz, the debug statements below are for the case when eth1
>>>>>>>>>>>is a
>>>>>>>>>>>control
>>>>>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>>>>>command,
>>>>>>>>>>> eth1 is not control, its either public or guest
>>>>>>>>>>>
>>>>>>>>>>> eth0: - control
>>>>>>>>>>>
>>>>>>>>>>> iface eth0 inet static
>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>
>>>>>>>>>>> eth1:
>>>>>>>>>>>
>>>>>>>>>>> iface eth1 inet static
>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>> Netmask
>>>>>>>>>>>
>>>>>>>>>>> So you need to execute the mysql statements for the traffic
>>>>>>>>>>>type
>>>>>>>>>>>of
>>>>>>>>>>>VR
>>>>>>>>>>>nic
>>>>>>>>>>> eth1
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>>>>>><Al...@citrix.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>>>>>
>>>>>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>>>>>traffic_type=‘Control’;
>>>>>>>>>>>>
>>>>>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>>>>>
>>>>>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid>
>>>>>>>>>>>>and
>>>>>>>>>>>>network_id=<control network id>
>>>>>>>>>>>>
>>>>>>>>>>>>-Alena.
>>>>>>>>>>>>
>>>>>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>I looked up the startup command of the old router instance
>>>>>>>>>>>>>which
>>>>>>>>>>>>>worked
>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>
>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomr
>>>>>>>>>>>>>pre
>>>>>>>>>>>>>.s
>>>>>>>>>>>>>h
>>>>>>>>>>>>>-l
>>>>>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.2
>>>>>>>>>>>>>55.
>>>>>>>>>>>>>25
>>>>>>>>>>>>>5.
>>>>>>>>>>>>>0%
>>>>>>>>>>>>>ga
>>>>>>>>>>>>>t
>>>>>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%dom
>>>>>>>>>>>>>ain
>>>>>>>>>>>>>=c
>>>>>>>>>>>>>s6
>>>>>>>>>>>>>cl
>>>>>>>>>>>>>ou
>>>>>>>>>>>>>d
>>>>>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=2
>>>>>>>>>>>>>55.
>>>>>>>>>>>>>25
>>>>>>>>>>>>>5.
>>>>>>>>>>>>>0.
>>>>>>>>>>>>>0%
>>>>>>>>>>>>>t
>>>>>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>
>>>>>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip
>>>>>>>>>>>>>and
>>>>>>>>>>>>>eth1mask.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>
>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>>>>>eliminated
>>>>>>>>>>>>>> static nat rule programming for non-existing vm. Missing
>>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>>on
>>>>>>>>>>>>>>eth1
>>>>>>>>>>>>>> doesn’t seem related to the problem (although we have to
>>>>>>>>>>>>>>figure
>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>why
>>>>>>>>>>>>>> its missing), as the connection that fails, happening to
>>>>>>>>>>>>>>link
>>>>>>>>>>>>>>local
>>>>>>>>>>>>>>169.x
>>>>>>>>>>>>>> eth0 interface.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Edison, can you please tell us how to debug link local
>>>>>>>>>>>>>>connection
>>>>>>>>>>>>>>failure,
>>>>>>>>>>>>>> on KVM agent?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>> Alena.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi"
>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more,
>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>When I start the virtual router, it comes up, but in
>>>>>>>>>>>>>>>agent.log,
>>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>>>>>...
>>>>>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>>>>> netmask
>>>>>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>>>>>agent.log
>>>>>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundo
>>>>>>>>>>>>>>>mrp
>>>>>>>>>>>>>>>re
>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>h
>>>>>>>>>>>>>>>-l
>>>>>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk
>>>>>>>>>>>>>>>-p
>>>>>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=25
>>>>>>>>>>>>>>>5.2
>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>>%
>>>>>>>>>>>>>>>ga
>>>>>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%
>>>>>>>>>>>>>>>dom
>>>>>>>>>>>>>>>ai
>>>>>>>>>>>>>>>n=
>>>>>>>>>>>>>>>cs
>>>>>>>>>>>>>>>6c
>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mas
>>>>>>>>>>>>>>>k=2
>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>>.
>>>>>>>>>>>>>>>0%
>>>>>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>>>>>culprit.
>>>>>>>>>>>>>>>>If
>>>>>>>>>>>>>>>>vm
>>>>>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to
>>>>>>>>>>>>>>>>it
>>>>>>>>>>>>>>>>from
>>>>>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0,
>>>>>>>>>>>>>>>>instance_id=null
>>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>>>>>following
>>>>>>>>>>>>>>>>>query
>>>>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules
>>>>>>>>>>>>>>>>>fr
>>>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+-
>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network
>>>>>>>>>>>>>>>>>>(205,
>>>>>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is
>>>>>>>>>>>>>>>>>>non-removed
>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static
>>>>>>>>>>>>>>>>>>nat
>>>>>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>>>>>> referring to vm that is removed already. If you find
>>>>>>>>>>>>>>>>>>any,
>>>>>>>>>>>>>>>>>>let
>>>>>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not
>>>>>>>>>>>>>>>>>>>null,
>>>>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router
>>>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was
>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>>>>>was used by my instances. I deleted the router using
>>>>>>>>>>>>>>>>>>>the UI
>>>>>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar
>>>>>>>>>>>>>>>>>>>problem
>>>>>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>>>>>that the deleted router would be recreated again (and
>>>>>>>>>>>>>>>>>>>this
>>>>>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 |
>>>>>>>>>>>>>>>>>>>255.255.255.0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+------------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+------
>>>>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>VPC
>>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to
>>>>>>>>>>>>>>>>>>>>revert
>>>>>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id
>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It will give you the list of networks refs that
>>>>>>>>>>>>>>>>>>>>somehow
>>>>>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to
>>>>>>>>>>>>>>>>>>>>>thank
>>>>>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote
>>>>>>>>>>>>>>>>>>>>>debugging, I
>>>>>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is
>>>>>>>>>>>>>>>>>>>>>not
>>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a
>>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress
>>>>>>>>>>>>>>>>>>>>>rule(s)
>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(Network
>>>>>>>>>>>>>>>>>>>>>Mo
>>>>>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>>>>>>>>>23
>>>>>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:7
>>>>>>>>>>>>>>>>>>>>>67
>>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtu
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachi
>>>>>>>>>>>>>>>>>>>>>ne
>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.ja
>>>>>>>>>>>>>>>>>>>>>va
>>>>>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(
>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElem
>>>>>>>>>>>>>>>>>>>>>en
>>>>>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Net
>>>>>>>>>>>>>>>>>>>>>wo
>>>>>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(Network
>>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRoute
>>>>>>>>>>>>>>>>>>>>>rC
>>>>>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>| id | uuid |
>>>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway |
>>>>>>>>>>>>>>>>>>>>>ip_type |
>>>>>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>>>> | removed | ip6_gateway |
>>>>>>>>>>>>>>>>>>>>>ip6_cidr |
>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |
>>>>>>>>>>>>>>>>>>>>>1 |
>>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |
>>>>>>>>>>>>>>>>>>>>>2 |
>>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |
>>>>>>>>>>>>>>>>>>>>>4
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |
>>>>>>>>>>>>>>>>>>>>>7
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |
>>>>>>>>>>>>>>>>>>>>>18
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |
>>>>>>>>>>>>>>>>>>>>>19
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
it gets even stranger than it was. After manually executing the iptables
command yesterday, I had the following in agent.log
http://pastie.org/8971483
and I could start vm instances, but couldn't ping them.
In the morning, I changed the device IDs
"nics":[
{"deviceId":2,
"defaultNic":true,
"ip":"10.193.17.190",
"type":"Public",
{"deviceId":1,
"ip":"10.124.99.1",
"type":"Guest",
{"deviceId":0,
"ip":"169.254.3.204",
"type":"Control",
but didn't observe any change.
Now (~ 15 hours later), I tried to start the virtual router and again
find the NullPointerException in agent.log (http://pastie.org/8971557):
2014-03-26 21:14:18,371 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-194) Failed to start instance VM[DomainRouter|r-19-VM]
java.lang.NullPointerException
at com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
So, we are back to square one. I ran this query:
select uip.vm_id, uip.network_id, uip.public_ip_address,
n.state as nic_state, n.removed as nic_removed,
vm.state as vm_state, vm.removed as vm_removed
from user_ip_address uip
join nics n on uip.vm_id = n.instance_id
join vm_instance vm on uip.vm_id = vm.id
where uip.id in (Select ip_address_id from firewall_rules fr
where
fr.network_id=205);
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
| vm_id | network_id | public_ip_address | nic_state | nic_removed | vm_state | vm_removed |
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
| 6 | 205 | 10.193.17.169 | Allocated | NULL | Stopped | NULL |
| 10 | 205 | 10.193.17.136 | Allocated | NULL | Stopped | NULL |
| 12 | 205 | 10.193.17.140 | Allocated | NULL | Stopped | NULL |
| 13 | 205 | 10.193.17.141 | Deallocating | 2014-03-26 07:03:27 | Expunging | NULL |
| 14 | 205 | 10.193.17.142 | Allocated | NULL | Stopped | NULL |
| 16 | 205 | 10.193.17.103 | Allocated | NULL | Stopped | NULL |
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
I know that I would have to run this, next:
update user_ip_address set one_to_one_nat=0, vm_id=null where vm_id=13;
But what are my options to get to a working system again? I'm struggling
with this issue since days.
Thank you
Kambiz
Kambiz Darabi <da...@m-creations.com> wrote:
>
> Yes, I know, but the messages do not stop, the router remains in state
> Starting for some time and is then shut down, because it couldn't be
> contacted.
>
> The problem is not yet completely solved. To make sshd listen all an
> interfaces, I did the following manually in the router VM:
>
> echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
> /etc/init.d/ssh restart
> iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 3922 -j ACCEPT
>
> After this, the router becomes 'Running' in the UI, agent.log says
>
> Ping command port succeeded for vm r-19-VM
> Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh get_template_version.sh 169.254.3.203
> Execution is successful.
>
> Maybe I set the wrong deviceId to 1 and should have taken the other nic
> with the deviceId 0. I will try that tomorrow and report back to you.
>
> Thank you so much for your help and patience.
>
>
>
> Kambiz
>
>
> Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> Seeing these messages for some time is fine as some delay in startup
>> process is expected, and eventually the command will succeed.
>>
>> telnet 169.254.3.155 3922
>> Trying 169.254.3.155…’
>>
>> So it works like this:
>>
>> 1) Start router is executed. Router is booting up
>> 2) Agent tries to ssh to the VR to ensure that the router is up
>> 3) Once its up, agent reports to the MS that startup was successful.
>>
>> As long as VR startup is reported as success to the MS, everything is good
>> and you shouldn’t worry about those messages.
>>
>> -Alena.
>>
>>
>>
>> On 3/25/14, 3:22 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>I executed
>>>
>>>> update nics set device_id = 1 where id = 29;
>>>
>>>After restarting the router, the interfaces file now looks like this:
>>>
>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>auto lo eth0 eth1 eth2
>>>iface lo inet loopback
>>>
>>>iface eth0 inet static
>>> address 169.254.3.155
>>> netmask 255.255.0.0
>>>iface eth1 inet static
>>> address 10.124.99.1
>>> netmask 255.255.255.0
>>>iface eth2 inet static
>>> address 10.193.17.190
>>> netmask 255.255.255.0
>>>
>>>ifconfig shows this:
>>>
>>>root@r-19-VM:~# ifconfig
>>>eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
>>> inet addr:169.254.3.155 Bcast:169.254.255.255 Mask:255.255.0.0
>>> inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>> RX packets:61 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:1000
>>> RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
>>>
>>>eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
>>> inet addr:10.124.99.1 Bcast:10.124.99.255 Mask:255.255.255.0
>>> inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>> RX packets:12 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:1000
>>> RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
>>>
>>>eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
>>> inet addr:10.193.17.190 Bcast:10.193.17.255 Mask:255.255.255.0
>>> inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>>> RX packets:11 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:1000
>>> RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>>>
>>>From inside the VM, I can ping the gateways of the public, guest and
>>>control network:
>>>
>>>root@r-19-VM:~# route -n
>>>Kernel IP routing table
>>>Destination Gateway Genmask Flags Metric Ref Use
>>>Iface
>>>10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>eth1
>>>10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0
>>>eth2
>>>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
>>>eth0
>>>0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0
>>>eth2
>>>
>>>root@r-19-VM:~# ping 10.193.17.1
>>>PING 10.193.17.1 (10.193.17.1): 56 data bytes
>>>64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
>>>64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
>>>
>>>root@r-19-VM:~# ping 10.124.99.1
>>>PING 10.124.99.1 (10.124.99.1): 56 data bytes
>>>64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
>>>
>>>root@r-19-VM:/etc/init.d# ping 169.254.0.1
>>>PING 169.254.0.1 (169.254.0.1): 56 data bytes
>>>64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
>>>
>>>And from outside, I can ping the different IPs of the router.
>>>
>>>But what is strange, is that in agent.log, I still find
>>>
>>>Ping command port, 169.254.3.155:3922
>>>Trying to connect to 169.254.3.155
>>>Could not connect to 169.254.3.155
>>>
>>>And when I check on the router, the ssh daemon only listens on the guest
>>>network interface:
>>>
>>># netstat -na | grep 3922
>>>tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
>>>
>>>So, the connection attempt to 169.254.3.155:3922 fails:
>>>
>>>telnet 169.254.3.155 3922
>>>Trying 169.254.3.155...
>>>
>>>Is that the normal situation?
>>>
>>>> Also after which point you started experiencing all this problems? Did
>>>> you upgrade to new CS version? Or does it fail for any specific network?
>>>
>>>No, I didn't upgrade to a new CS, I just stopped and started the
>>>management-server.
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> Kambiz, did you check the device id in nics table for the vm? If it has
>>>>2
>>>> 0s, change one of them to the correct value and restart. If the testing
>>>> completes fine, we have a proof that its related to the device id mix
>>>>up.
>>>> If not, there is gotta be something else, most likely misconfigured on
>>>>KVM
>>>> stuff.
>>>>
>>>> Also after which point you started experiencing all this problems? Did
>>>> you upgrade to new CS version? Or does it fail for any specific network?
>>>>
>>>> -Alena.
>>>>
>>>> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>I updated nics.gateway for that network, but the VM still shows the same
>>>>>behaviour.
>>>>>
>>>>>If one compares interfaces:
>>>>>
>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>auto lo eth0 eth1 eth2
>>>>>iface lo inet loopback
>>>>>
>>>>>iface eth0 inet static
>>>>> address 169.254.1.242
>>>>> netmask 255.255.0.0
>>>>>iface eth1 inet static
>>>>> address 10.193.17.1
>>>>> netmask
>>>>>iface eth2 inet static
>>>>> address 10.193.17.190
>>>>> netmask 255.255.255.0
>>>>>
>>>>>and the nics entry in the management-server.log (cf. below), one can see
>>>>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
>>>>>correctly set up with IP 10.193.17.190.
>>>>>
>>>>>The remaining nic is eth1 which corresponds to the first nic with
>>>>>deviceId 0 and according to the nics entry should have IP 10.124.99.1,
>>>>>but there is no iface entry with that IP, but eth1 has the gateway
>>>>>address of the public nic as its IP address.
>>>>>
>>>>>Could the problem have something to do with the duplicate deviceId 0?
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>{ "nics":
>>>>> [{"deviceId":2,
>>>>> "networkRateMbps":200,
>>>>> "defaultNic":true,
>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>> "ip":"10.193.17.190",
>>>>> "netmask":"255.255.255.0",
>>>>> "gateway":"10.193.17.1",
>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>> "dns1":"10.193.17.1",
>>>>> "broadcastType":"Vlan",
>>>>> "type":"Public",
>>>>> "broadcastUri":"vlan://untagged",
>>>>> "isolationUri":"vlan://untagged",
>>>>> "isSecurityGroupEnabled":false,
>>>>> "name":"cloudbr0"},
>>>>> {"deviceId":0,
>>>>> "networkRateMbps":200,
>>>>> "defaultNic":false,
>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>> "ip":"10.124.99.1",
>>>>> "netmask":"255.255.255.0",
>>>>> "gateway":"10.124.99.1",
>>>>> "mac":"02:00:2a:43:00:0d",
>>>>> "dns1":"10.193.17.1",
>>>>> "broadcastType":"Vlan",
>>>>> "type":"Guest",
>>>>> "broadcastUri":"vlan://3925",
>>>>> "isolationUri":"vlan://3925",
>>>>> "isSecurityGroupEnabled":false,
>>>>> "name":"cloudbr1"},
>>>>> {"deviceId":0,
>>>>> "networkRateMbps":-1,
>>>>> "defaultNic":false,
>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>> "ip":"169.254.1.242",
>>>>> "netmask":"255.255.0.0",
>>>>> "gateway":"169.254.0.1",
>>>>> "mac":"0e:00:a9:fe:01:f2",
>>>>> "broadcastType":"LinkLocal",
>>>>> "type":"Control",
>>>>> "isSecurityGroupEnabled":false}
>>>>> ]
>>>>>}
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> So the gateway wasn’t set for the nic only.
>>>>>>
>>>>>> Kambiz, just to quickly test it, can you set missing gateway for the
>>>>>>nic,
>>>>>> and stop/start the VR? And see if the start is completed normally,
>>>>>>just
>>>>>>to
>>>>>> find out if the missing gateway was the reason of the communication
>>>>>>failure
>>>>>>
>>>>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>>>>>>mode, state, removed from networks where id = 205;
>>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>>--
>>>>>>>+-
>>>>>>>------------+------+-----------+---------+
>>>>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>>>>|
>>>>>>>gateway | mode | state | removed |
>>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>>--
>>>>>>>+-
>>>>>>>------------+------+-----------+---------+
>>>>>>>| 205 | default | Guest | Vlan |
>>>>>>>10.124.99.0/24
>>>>>>>|
>>>>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>>--
>>>>>>>+-
>>>>>>>------------+------+-----------+---------+
>>>>>>>
>>>>>>>Cheers
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>>>>>>looks
>>>>>>>> like they’ve got programmed to correct devices on the backend per
>>>>>>>>your
>>>>>>>> prev email?
>>>>>>>>
>>>>>>>> iface eth0 inet static
>>>>>>>> address 169.254.1.59
>>>>>>>> netmask 255.255.0.0
>>>>>>>>
>>>>>>>> iface eth1 inet static
>>>>>>>> address 10.193.17.1
>>>>>>>> Netmask
>>>>>>>>
>>>>>>>> iface eth2 inet static
>>>>>>>> address 10.193.17.190
>>>>>>>> netmask 255.255.255.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I can see that only one parameter is missing from the start command,
>>>>>>>>the
>>>>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>>>>> From the command/DB, I see that the gateway is missing in the nics
>>>>>>>>table
>>>>>>>> for the network 205? Can you check gateway information in the
>>>>>>>>networks
>>>>>>>> table for the id=205
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>select
>>>>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_nam
>>>>>>>>>e
>>>>>>>>>from nics where instance_id=19;
>>>>>>>>>
>>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>------------+--------------------------+
>>>>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>>>>removed
>>>>>>>>>| network_id | reserver_name |
>>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>------------+--------------------------+
>>>>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated |
>>>>>>>>>NULL
>>>>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>>>>| 30 | NULL | NULL | NULL | Allocated |
>>>>>>>>>NULL
>>>>>>>>>| 202 | ControlNetworkGuru |
>>>>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated |
>>>>>>>>>NULL
>>>>>>>>>| 200 | PublicNetworkGuru |
>>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>------------+--------------------------+
>>>>>>>>>
>>>>>>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>>>>>>two nics with deviceId 0?
>>>>>>>>>
>>>>>>>>>"nics":[
>>>>>>>>> {"deviceId":2,
>>>>>>>>> "networkRateMbps":200,
>>>>>>>>> "defaultNic":true,
>>>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>>>> "ip":"10.193.17.190",
>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>> "gateway":"10.193.17.1",
>>>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>> "type":"Public",
>>>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>> "name":"cloudbr0"},
>>>>>>>>> {"deviceId":0,
>>>>>>>>> "networkRateMbps":200,
>>>>>>>>> "defaultNic":false,
>>>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>>>> "ip":"10.124.99.1",
>>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>>> "broadcastType":"Vlan",
>>>>>>>>> "type":"Guest",
>>>>>>>>> "broadcastUri":"vlan://3949",
>>>>>>>>> "isolationUri":"vlan://3949",
>>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>>> "name":"cloudbr1"},
>>>>>>>>> {"deviceId":0,
>>>>>>>>> "networkRateMbps":-1,
>>>>>>>>> "defaultNic":false,
>>>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>>>> "ip":"169.254.1.59",
>>>>>>>>> "netmask":"255.255.0.0",
>>>>>>>>> "gateway":"169.254.0.1",
>>>>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>>>>> "broadcastType":"LinkLocal",
>>>>>>>>> "type":"Control",
>>>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>>>>]
>>>>>>>>>
>>>>>>>>>Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>>>>>>control
>>>>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>>>>command,
>>>>>>>>>> eth1 is not control, its either public or guest
>>>>>>>>>>
>>>>>>>>>> eth0: - control
>>>>>>>>>>
>>>>>>>>>> iface eth0 inet static
>>>>>>>>>> address 169.254.1.59
>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>
>>>>>>>>>> eth1:
>>>>>>>>>>
>>>>>>>>>> iface eth1 inet static
>>>>>>>>>> address 10.193.17.1
>>>>>>>>>> Netmask
>>>>>>>>>>
>>>>>>>>>> So you need to execute the mysql statements for the traffic type
>>>>>>>>>>of
>>>>>>>>>>VR
>>>>>>>>>>nic
>>>>>>>>>> eth1
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>>>>><Al...@citrix.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>>>>
>>>>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>>>>traffic_type=‘Control’;
>>>>>>>>>>>
>>>>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>>>>
>>>>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>>>>>>network_id=<control network id>
>>>>>>>>>>>
>>>>>>>>>>>-Alena.
>>>>>>>>>>>
>>>>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>>>>>>worked
>>>>>>>>>>>>correctly:
>>>>>>>>>>>>
>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre
>>>>>>>>>>>>.s
>>>>>>>>>>>>h
>>>>>>>>>>>>-l
>>>>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.
>>>>>>>>>>>>25
>>>>>>>>>>>>5.
>>>>>>>>>>>>0%
>>>>>>>>>>>>ga
>>>>>>>>>>>>t
>>>>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain
>>>>>>>>>>>>=c
>>>>>>>>>>>>s6
>>>>>>>>>>>>cl
>>>>>>>>>>>>ou
>>>>>>>>>>>>d
>>>>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.
>>>>>>>>>>>>25
>>>>>>>>>>>>5.
>>>>>>>>>>>>0.
>>>>>>>>>>>>0%
>>>>>>>>>>>>t
>>>>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>
>>>>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip
>>>>>>>>>>>>and
>>>>>>>>>>>>eth1mask.
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>
>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>>>>eliminated
>>>>>>>>>>>>> static nat rule programming for non-existing vm. Missing
>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>on
>>>>>>>>>>>>>eth1
>>>>>>>>>>>>> doesn’t seem related to the problem (although we have to figure
>>>>>>>>>>>>>out
>>>>>>>>>>>>>why
>>>>>>>>>>>>> its missing), as the connection that fails, happening to link
>>>>>>>>>>>>>local
>>>>>>>>>>>>>169.x
>>>>>>>>>>>>> eth0 interface.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Edison, can you please tell us how to debug link local
>>>>>>>>>>>>>connection
>>>>>>>>>>>>>failure,
>>>>>>>>>>>>> on KVM agent?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>> Alena.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>>>>>>there
>>>>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>>>>...
>>>>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>>>> netmask
>>>>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>>>>agent.log
>>>>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrp
>>>>>>>>>>>>>>re
>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>h
>>>>>>>>>>>>>>-l
>>>>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.2
>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>%
>>>>>>>>>>>>>>ga
>>>>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%dom
>>>>>>>>>>>>>>ai
>>>>>>>>>>>>>>n=
>>>>>>>>>>>>>>cs
>>>>>>>>>>>>>>6c
>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=2
>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>.2
>>>>>>>>>>>>>>55
>>>>>>>>>>>>>>.0
>>>>>>>>>>>>>>.
>>>>>>>>>>>>>>0%
>>>>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>>>>culprit.
>>>>>>>>>>>>>>>If
>>>>>>>>>>>>>>>vm
>>>>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it
>>>>>>>>>>>>>>>from
>>>>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>>>>following
>>>>>>>>>>>>>>>>query
>>>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network
>>>>>>>>>>>>>>>>>(205,
>>>>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is
>>>>>>>>>>>>>>>>>non-removed
>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat
>>>>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>>>>> referring to vm that is removed already. If you find any,
>>>>>>>>>>>>>>>>>let
>>>>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null,
>>>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router
>>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar
>>>>>>>>>>>>>>>>>>problem
>>>>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because the
>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in
>>>>>>>>>>>>>>>>>>>VPC
>>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to
>>>>>>>>>>>>>>>>>>>revert
>>>>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of
>>>>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to
>>>>>>>>>>>>>>>>>>>>thank
>>>>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote
>>>>>>>>>>>>>>>>>>>>debugging, I
>>>>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is
>>>>>>>>>>>>>>>>>>>>not
>>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a
>>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress
>>>>>>>>>>>>>>>>>>>>rule(s)
>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(Network
>>>>>>>>>>>>>>>>>>>>Mo
>>>>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>>>>>>>>23
>>>>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:7
>>>>>>>>>>>>>>>>>>>>67
>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtu
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachi
>>>>>>>>>>>>>>>>>>>>ne
>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.ja
>>>>>>>>>>>>>>>>>>>>va
>>>>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(
>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElem
>>>>>>>>>>>>>>>>>>>>en
>>>>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Net
>>>>>>>>>>>>>>>>>>>>wo
>>>>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(Network
>>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRoute
>>>>>>>>>>>>>>>>>>>>rC
>>>>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>| id | uuid |
>>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway |
>>>>>>>>>>>>>>>>>>>>ip_type |
>>>>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>>> | removed | ip6_gateway |
>>>>>>>>>>>>>>>>>>>>ip6_cidr |
>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |
>>>>>>>>>>>>>>>>>>>>1 |
>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |
>>>>>>>>>>>>>>>>>>>>2 |
>>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |
>>>>>>>>>>>>>>>>>>>>4
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |
>>>>>>>>>>>>>>>>>>>>7
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |
>>>>>>>>>>>>>>>>>>>>18
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |
>>>>>>>>>>>>>>>>>>>>19
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL
>>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Yes, I know, but the messages do not stop, the router remains in state
Starting for some time and is then shut down, because it couldn't be
contacted.
The problem is not yet completely solved. To make sshd listen all an
interfaces, I did the following manually in the router VM:
echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
/etc/init.d/ssh restart
iptables -A INPUT -i eth0 -p tcp -m state --state NEW --dport 3922 -j ACCEPT
After this, the router becomes 'Running' in the UI, agent.log says
Ping command port succeeded for vm r-19-VM
Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh get_template_version.sh 169.254.3.203
Execution is successful.
Maybe I set the wrong deviceId to 1 and should have taken the other nic
with the deviceId 0. I will try that tomorrow and report back to you.
Thank you so much for your help and patience.
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Seeing these messages for some time is fine as some delay in startup
> process is expected, and eventually the command will succeed.
>
> telnet 169.254.3.155 3922
> Trying 169.254.3.155…’
>
> So it works like this:
>
> 1) Start router is executed. Router is booting up
> 2) Agent tries to ssh to the VR to ensure that the router is up
> 3) Once its up, agent reports to the MS that startup was successful.
>
> As long as VR startup is reported as success to the MS, everything is good
> and you shouldn’t worry about those messages.
>
> -Alena.
>
>
>
> On 3/25/14, 3:22 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>I executed
>>
>>> update nics set device_id = 1 where id = 29;
>>
>>After restarting the router, the interfaces file now looks like this:
>>
>>root@r-19-VM:~# cat /etc/network/interfaces
>>auto lo eth0 eth1 eth2
>>iface lo inet loopback
>>
>>iface eth0 inet static
>> address 169.254.3.155
>> netmask 255.255.0.0
>>iface eth1 inet static
>> address 10.124.99.1
>> netmask 255.255.255.0
>>iface eth2 inet static
>> address 10.193.17.190
>> netmask 255.255.255.0
>>
>>ifconfig shows this:
>>
>>root@r-19-VM:~# ifconfig
>>eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
>> inet addr:169.254.3.155 Bcast:169.254.255.255 Mask:255.255.0.0
>> inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:61 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
>>
>>eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
>> inet addr:10.124.99.1 Bcast:10.124.99.255 Mask:255.255.255.0
>> inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:12 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
>>
>>eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
>> inet addr:10.193.17.190 Bcast:10.193.17.255 Mask:255.255.255.0
>> inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:11 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>>
>>From inside the VM, I can ping the gateways of the public, guest and
>>control network:
>>
>>root@r-19-VM:~# route -n
>>Kernel IP routing table
>>Destination Gateway Genmask Flags Metric Ref Use
>>Iface
>>10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0
>>eth1
>>10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0
>>eth2
>>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
>>eth0
>>0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0
>>eth2
>>
>>root@r-19-VM:~# ping 10.193.17.1
>>PING 10.193.17.1 (10.193.17.1): 56 data bytes
>>64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
>>64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
>>
>>root@r-19-VM:~# ping 10.124.99.1
>>PING 10.124.99.1 (10.124.99.1): 56 data bytes
>>64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
>>
>>root@r-19-VM:/etc/init.d# ping 169.254.0.1
>>PING 169.254.0.1 (169.254.0.1): 56 data bytes
>>64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
>>
>>And from outside, I can ping the different IPs of the router.
>>
>>But what is strange, is that in agent.log, I still find
>>
>>Ping command port, 169.254.3.155:3922
>>Trying to connect to 169.254.3.155
>>Could not connect to 169.254.3.155
>>
>>And when I check on the router, the ssh daemon only listens on the guest
>>network interface:
>>
>># netstat -na | grep 3922
>>tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
>>
>>So, the connection attempt to 169.254.3.155:3922 fails:
>>
>>telnet 169.254.3.155 3922
>>Trying 169.254.3.155...
>>
>>Is that the normal situation?
>>
>>> Also after which point you started experiencing all this problems? Did
>>> you upgrade to new CS version? Or does it fail for any specific network?
>>
>>No, I didn't upgrade to a new CS, I just stopped and started the
>>management-server.
>>
>>Thanks
>>
>>
>>Kambiz
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> Kambiz, did you check the device id in nics table for the vm? If it has
>>>2
>>> 0s, change one of them to the correct value and restart. If the testing
>>> completes fine, we have a proof that its related to the device id mix
>>>up.
>>> If not, there is gotta be something else, most likely misconfigured on
>>>KVM
>>> stuff.
>>>
>>> Also after which point you started experiencing all this problems? Did
>>> you upgrade to new CS version? Or does it fail for any specific network?
>>>
>>> -Alena.
>>>
>>> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>I updated nics.gateway for that network, but the VM still shows the same
>>>>behaviour.
>>>>
>>>>If one compares interfaces:
>>>>
>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>auto lo eth0 eth1 eth2
>>>>iface lo inet loopback
>>>>
>>>>iface eth0 inet static
>>>> address 169.254.1.242
>>>> netmask 255.255.0.0
>>>>iface eth1 inet static
>>>> address 10.193.17.1
>>>> netmask
>>>>iface eth2 inet static
>>>> address 10.193.17.190
>>>> netmask 255.255.255.0
>>>>
>>>>and the nics entry in the management-server.log (cf. below), one can see
>>>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
>>>>correctly set up with IP 10.193.17.190.
>>>>
>>>>The remaining nic is eth1 which corresponds to the first nic with
>>>>deviceId 0 and according to the nics entry should have IP 10.124.99.1,
>>>>but there is no iface entry with that IP, but eth1 has the gateway
>>>>address of the public nic as its IP address.
>>>>
>>>>Could the problem have something to do with the duplicate deviceId 0?
>>>>
>>>>Thanks
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>{ "nics":
>>>> [{"deviceId":2,
>>>> "networkRateMbps":200,
>>>> "defaultNic":true,
>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>> "ip":"10.193.17.190",
>>>> "netmask":"255.255.255.0",
>>>> "gateway":"10.193.17.1",
>>>> "mac":"06:7e:fe:00:00:bf",
>>>> "dns1":"10.193.17.1",
>>>> "broadcastType":"Vlan",
>>>> "type":"Public",
>>>> "broadcastUri":"vlan://untagged",
>>>> "isolationUri":"vlan://untagged",
>>>> "isSecurityGroupEnabled":false,
>>>> "name":"cloudbr0"},
>>>> {"deviceId":0,
>>>> "networkRateMbps":200,
>>>> "defaultNic":false,
>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>> "ip":"10.124.99.1",
>>>> "netmask":"255.255.255.0",
>>>> "gateway":"10.124.99.1",
>>>> "mac":"02:00:2a:43:00:0d",
>>>> "dns1":"10.193.17.1",
>>>> "broadcastType":"Vlan",
>>>> "type":"Guest",
>>>> "broadcastUri":"vlan://3925",
>>>> "isolationUri":"vlan://3925",
>>>> "isSecurityGroupEnabled":false,
>>>> "name":"cloudbr1"},
>>>> {"deviceId":0,
>>>> "networkRateMbps":-1,
>>>> "defaultNic":false,
>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>> "ip":"169.254.1.242",
>>>> "netmask":"255.255.0.0",
>>>> "gateway":"169.254.0.1",
>>>> "mac":"0e:00:a9:fe:01:f2",
>>>> "broadcastType":"LinkLocal",
>>>> "type":"Control",
>>>> "isSecurityGroupEnabled":false}
>>>> ]
>>>>}
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> So the gateway wasn’t set for the nic only.
>>>>>
>>>>> Kambiz, just to quickly test it, can you set missing gateway for the
>>>>>nic,
>>>>> and stop/start the VR? And see if the start is completed normally,
>>>>>just
>>>>>to
>>>>> find out if the missing gateway was the reason of the communication
>>>>>failure
>>>>>
>>>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>>>>>mode, state, removed from networks where id = 205;
>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>--
>>>>>>+-
>>>>>>------------+------+-----------+---------+
>>>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>>>|
>>>>>>gateway | mode | state | removed |
>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>--
>>>>>>+-
>>>>>>------------+------+-----------+---------+
>>>>>>| 205 | default | Guest | Vlan |
>>>>>>10.124.99.0/24
>>>>>>|
>>>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>>--
>>>>>>+-
>>>>>>------------+------+-----------+---------+
>>>>>>
>>>>>>Cheers
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>>>>>looks
>>>>>>> like they’ve got programmed to correct devices on the backend per
>>>>>>>your
>>>>>>> prev email?
>>>>>>>
>>>>>>> iface eth0 inet static
>>>>>>> address 169.254.1.59
>>>>>>> netmask 255.255.0.0
>>>>>>>
>>>>>>> iface eth1 inet static
>>>>>>> address 10.193.17.1
>>>>>>> Netmask
>>>>>>>
>>>>>>> iface eth2 inet static
>>>>>>> address 10.193.17.190
>>>>>>> netmask 255.255.255.0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I can see that only one parameter is missing from the start command,
>>>>>>>the
>>>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>>>> From the command/DB, I see that the gateway is missing in the nics
>>>>>>>table
>>>>>>> for the network 205? Can you check gateway information in the
>>>>>>>networks
>>>>>>> table for the id=205
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>select
>>>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_nam
>>>>>>>>e
>>>>>>>>from nics where instance_id=19;
>>>>>>>>
>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>------------+--------------------------+
>>>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>>>removed
>>>>>>>>| network_id | reserver_name |
>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>------------+--------------------------+
>>>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated |
>>>>>>>>NULL
>>>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>>>| 30 | NULL | NULL | NULL | Allocated |
>>>>>>>>NULL
>>>>>>>>| 202 | ControlNetworkGuru |
>>>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated |
>>>>>>>>NULL
>>>>>>>>| 200 | PublicNetworkGuru |
>>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>------------+--------------------------+
>>>>>>>>
>>>>>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>>>>>two nics with deviceId 0?
>>>>>>>>
>>>>>>>>"nics":[
>>>>>>>> {"deviceId":2,
>>>>>>>> "networkRateMbps":200,
>>>>>>>> "defaultNic":true,
>>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>>> "ip":"10.193.17.190",
>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>> "gateway":"10.193.17.1",
>>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>> "broadcastType":"Vlan",
>>>>>>>> "type":"Public",
>>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>> "name":"cloudbr0"},
>>>>>>>> {"deviceId":0,
>>>>>>>> "networkRateMbps":200,
>>>>>>>> "defaultNic":false,
>>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>>> "ip":"10.124.99.1",
>>>>>>>> "netmask":"255.255.255.0",
>>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>>> "dns1":"10.193.17.1",
>>>>>>>> "broadcastType":"Vlan",
>>>>>>>> "type":"Guest",
>>>>>>>> "broadcastUri":"vlan://3949",
>>>>>>>> "isolationUri":"vlan://3949",
>>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>>> "name":"cloudbr1"},
>>>>>>>> {"deviceId":0,
>>>>>>>> "networkRateMbps":-1,
>>>>>>>> "defaultNic":false,
>>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>>> "ip":"169.254.1.59",
>>>>>>>> "netmask":"255.255.0.0",
>>>>>>>> "gateway":"169.254.0.1",
>>>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>>>> "broadcastType":"LinkLocal",
>>>>>>>> "type":"Control",
>>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>>>]
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>>>>>control
>>>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>>>command,
>>>>>>>>> eth1 is not control, its either public or guest
>>>>>>>>>
>>>>>>>>> eth0: - control
>>>>>>>>>
>>>>>>>>> iface eth0 inet static
>>>>>>>>> address 169.254.1.59
>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>
>>>>>>>>> eth1:
>>>>>>>>>
>>>>>>>>> iface eth1 inet static
>>>>>>>>> address 10.193.17.1
>>>>>>>>> Netmask
>>>>>>>>>
>>>>>>>>> So you need to execute the mysql statements for the traffic type
>>>>>>>>>of
>>>>>>>>>VR
>>>>>>>>>nic
>>>>>>>>> eth1
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>>>><Al...@citrix.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>>>
>>>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>>>traffic_type=‘Control’;
>>>>>>>>>>
>>>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>>>
>>>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>>>>>network_id=<control network id>
>>>>>>>>>>
>>>>>>>>>>-Alena.
>>>>>>>>>>
>>>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi,
>>>>>>>>>>>
>>>>>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>>>>>worked
>>>>>>>>>>>correctly:
>>>>>>>>>>>
>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre
>>>>>>>>>>>.s
>>>>>>>>>>>h
>>>>>>>>>>>-l
>>>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.
>>>>>>>>>>>25
>>>>>>>>>>>5.
>>>>>>>>>>>0%
>>>>>>>>>>>ga
>>>>>>>>>>>t
>>>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain
>>>>>>>>>>>=c
>>>>>>>>>>>s6
>>>>>>>>>>>cl
>>>>>>>>>>>ou
>>>>>>>>>>>d
>>>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.
>>>>>>>>>>>25
>>>>>>>>>>>5.
>>>>>>>>>>>0.
>>>>>>>>>>>0%
>>>>>>>>>>>t
>>>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>
>>>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip
>>>>>>>>>>>and
>>>>>>>>>>>eth1mask.
>>>>>>>>>>>
>>>>>>>>>>>Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>>>eliminated
>>>>>>>>>>>> static nat rule programming for non-existing vm. Missing
>>>>>>>>>>>>netmask
>>>>>>>>>>>>on
>>>>>>>>>>>>eth1
>>>>>>>>>>>> doesn’t seem related to the problem (although we have to figure
>>>>>>>>>>>>out
>>>>>>>>>>>>why
>>>>>>>>>>>> its missing), as the connection that fails, happening to link
>>>>>>>>>>>>local
>>>>>>>>>>>>169.x
>>>>>>>>>>>> eth0 interface.
>>>>>>>>>>>>
>>>>>>>>>>>> Edison, can you please tell us how to debug link local
>>>>>>>>>>>>connection
>>>>>>>>>>>>failure,
>>>>>>>>>>>> on KVM agent?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>> Alena.
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>>>>>there
>>>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>>>
>>>>>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>>>>>there
>>>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>>>netmask
>>>>>>>>>>>>>of
>>>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>>>
>>>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>>>
>>>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>>>
>>>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>>>...
>>>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>>>
>>>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>>> netmask
>>>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>>>
>>>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>>>agent.log
>>>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>>>
>>>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrp
>>>>>>>>>>>>>re
>>>>>>>>>>>>>.s
>>>>>>>>>>>>>h
>>>>>>>>>>>>>-l
>>>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.2
>>>>>>>>>>>>>55
>>>>>>>>>>>>>.2
>>>>>>>>>>>>>55
>>>>>>>>>>>>>.0
>>>>>>>>>>>>>%
>>>>>>>>>>>>>ga
>>>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%dom
>>>>>>>>>>>>>ai
>>>>>>>>>>>>>n=
>>>>>>>>>>>>>cs
>>>>>>>>>>>>>6c
>>>>>>>>>>>>>l
>>>>>>>>>>>>>ou
>>>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=2
>>>>>>>>>>>>>55
>>>>>>>>>>>>>.2
>>>>>>>>>>>>>55
>>>>>>>>>>>>>.0
>>>>>>>>>>>>>.
>>>>>>>>>>>>>0%
>>>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>>>culprit.
>>>>>>>>>>>>>>If
>>>>>>>>>>>>>>vm
>>>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it
>>>>>>>>>>>>>>from
>>>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>>>>>where
>>>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi"
>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>>>following
>>>>>>>>>>>>>>>query
>>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network
>>>>>>>>>>>>>>>>(205,
>>>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is
>>>>>>>>>>>>>>>>non-removed
>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat
>>>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>>>> referring to vm that is removed already. If you find any,
>>>>>>>>>>>>>>>>let
>>>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null,
>>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router
>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar
>>>>>>>>>>>>>>>>>problem
>>>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because the
>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in
>>>>>>>>>>>>>>>>>>VPC
>>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to
>>>>>>>>>>>>>>>>>>revert
>>>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of
>>>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to
>>>>>>>>>>>>>>>>>>>thank
>>>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote
>>>>>>>>>>>>>>>>>>>debugging, I
>>>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is
>>>>>>>>>>>>>>>>>>>not
>>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a
>>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress
>>>>>>>>>>>>>>>>>>>rule(s)
>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(Network
>>>>>>>>>>>>>>>>>>>Mo
>>>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>>>>>>>23
>>>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:7
>>>>>>>>>>>>>>>>>>>67
>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtu
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachi
>>>>>>>>>>>>>>>>>>>ne
>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.ja
>>>>>>>>>>>>>>>>>>>va
>>>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(
>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElem
>>>>>>>>>>>>>>>>>>>en
>>>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Net
>>>>>>>>>>>>>>>>>>>wo
>>>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(Network
>>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRoute
>>>>>>>>>>>>>>>>>>>rC
>>>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>| id | uuid |
>>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway |
>>>>>>>>>>>>>>>>>>>ip_type |
>>>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>>> | removed | ip6_gateway |
>>>>>>>>>>>>>>>>>>>ip6_cidr |
>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |
>>>>>>>>>>>>>>>>>>>1 |
>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |
>>>>>>>>>>>>>>>>>>>2 |
>>>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |
>>>>>>>>>>>>>>>>>>>4
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |
>>>>>>>>>>>>>>>>>>>7
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |
>>>>>>>>>>>>>>>>>>>18
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |
>>>>>>>>>>>>>>>>>>>19
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL
>>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Seeing these messages for some time is fine as some delay in startup
process is expected, and eventually the command will succeed.
telnet 169.254.3.155 3922
Trying 169.254.3.155…’
So it works like this:
1) Start router is executed. Router is booting up
2) Agent tries to ssh to the VR to ensure that the router is up
3) Once its up, agent reports to the MS that startup was successful.
As long as VR startup is reported as success to the MS, everything is good
and you shouldn’t worry about those messages.
-Alena.
On 3/25/14, 3:22 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>I executed
>
>> update nics set device_id = 1 where id = 29;
>
>After restarting the router, the interfaces file now looks like this:
>
>root@r-19-VM:~# cat /etc/network/interfaces
>auto lo eth0 eth1 eth2
>iface lo inet loopback
>
>iface eth0 inet static
> address 169.254.3.155
> netmask 255.255.0.0
>iface eth1 inet static
> address 10.124.99.1
> netmask 255.255.255.0
>iface eth2 inet static
> address 10.193.17.190
> netmask 255.255.255.0
>
>ifconfig shows this:
>
>root@r-19-VM:~# ifconfig
>eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
> inet addr:169.254.3.155 Bcast:169.254.255.255 Mask:255.255.0.0
> inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:61 errors:0 dropped:0 overruns:0 frame:0
> TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
>
>eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
> inet addr:10.124.99.1 Bcast:10.124.99.255 Mask:255.255.255.0
> inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:12 errors:0 dropped:0 overruns:0 frame:0
> TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
>
>eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
> inet addr:10.193.17.190 Bcast:10.193.17.255 Mask:255.255.255.0
> inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:11 errors:0 dropped:0 overruns:0 frame:0
> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>
>From inside the VM, I can ping the gateways of the public, guest and
>control network:
>
>root@r-19-VM:~# route -n
>Kernel IP routing table
>Destination Gateway Genmask Flags Metric Ref Use
>Iface
>10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0
>eth1
>10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0
>eth2
>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
>eth0
>0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0
>eth2
>
>root@r-19-VM:~# ping 10.193.17.1
>PING 10.193.17.1 (10.193.17.1): 56 data bytes
>64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
>64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
>
>root@r-19-VM:~# ping 10.124.99.1
>PING 10.124.99.1 (10.124.99.1): 56 data bytes
>64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
>
>root@r-19-VM:/etc/init.d# ping 169.254.0.1
>PING 169.254.0.1 (169.254.0.1): 56 data bytes
>64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
>
>And from outside, I can ping the different IPs of the router.
>
>But what is strange, is that in agent.log, I still find
>
>Ping command port, 169.254.3.155:3922
>Trying to connect to 169.254.3.155
>Could not connect to 169.254.3.155
>
>And when I check on the router, the ssh daemon only listens on the guest
>network interface:
>
># netstat -na | grep 3922
>tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
>
>So, the connection attempt to 169.254.3.155:3922 fails:
>
>telnet 169.254.3.155 3922
>Trying 169.254.3.155...
>
>Is that the normal situation?
>
>> Also after which point you started experiencing all this problems? Did
>> you upgrade to new CS version? Or does it fail for any specific network?
>
>No, I didn't upgrade to a new CS, I just stopped and started the
>management-server.
>
>Thanks
>
>
>Kambiz
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> Kambiz, did you check the device id in nics table for the vm? If it has
>>2
>> 0s, change one of them to the correct value and restart. If the testing
>> completes fine, we have a proof that its related to the device id mix
>>up.
>> If not, there is gotta be something else, most likely misconfigured on
>>KVM
>> stuff.
>>
>> Also after which point you started experiencing all this problems? Did
>> you upgrade to new CS version? Or does it fail for any specific network?
>>
>> -Alena.
>>
>> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>I updated nics.gateway for that network, but the VM still shows the same
>>>behaviour.
>>>
>>>If one compares interfaces:
>>>
>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>auto lo eth0 eth1 eth2
>>>iface lo inet loopback
>>>
>>>iface eth0 inet static
>>> address 169.254.1.242
>>> netmask 255.255.0.0
>>>iface eth1 inet static
>>> address 10.193.17.1
>>> netmask
>>>iface eth2 inet static
>>> address 10.193.17.190
>>> netmask 255.255.255.0
>>>
>>>and the nics entry in the management-server.log (cf. below), one can see
>>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
>>>correctly set up with IP 10.193.17.190.
>>>
>>>The remaining nic is eth1 which corresponds to the first nic with
>>>deviceId 0 and according to the nics entry should have IP 10.124.99.1,
>>>but there is no iface entry with that IP, but eth1 has the gateway
>>>address of the public nic as its IP address.
>>>
>>>Could the problem have something to do with the duplicate deviceId 0?
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>{ "nics":
>>> [{"deviceId":2,
>>> "networkRateMbps":200,
>>> "defaultNic":true,
>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>> "ip":"10.193.17.190",
>>> "netmask":"255.255.255.0",
>>> "gateway":"10.193.17.1",
>>> "mac":"06:7e:fe:00:00:bf",
>>> "dns1":"10.193.17.1",
>>> "broadcastType":"Vlan",
>>> "type":"Public",
>>> "broadcastUri":"vlan://untagged",
>>> "isolationUri":"vlan://untagged",
>>> "isSecurityGroupEnabled":false,
>>> "name":"cloudbr0"},
>>> {"deviceId":0,
>>> "networkRateMbps":200,
>>> "defaultNic":false,
>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>> "ip":"10.124.99.1",
>>> "netmask":"255.255.255.0",
>>> "gateway":"10.124.99.1",
>>> "mac":"02:00:2a:43:00:0d",
>>> "dns1":"10.193.17.1",
>>> "broadcastType":"Vlan",
>>> "type":"Guest",
>>> "broadcastUri":"vlan://3925",
>>> "isolationUri":"vlan://3925",
>>> "isSecurityGroupEnabled":false,
>>> "name":"cloudbr1"},
>>> {"deviceId":0,
>>> "networkRateMbps":-1,
>>> "defaultNic":false,
>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>> "ip":"169.254.1.242",
>>> "netmask":"255.255.0.0",
>>> "gateway":"169.254.0.1",
>>> "mac":"0e:00:a9:fe:01:f2",
>>> "broadcastType":"LinkLocal",
>>> "type":"Control",
>>> "isSecurityGroupEnabled":false}
>>> ]
>>>}
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> So the gateway wasn’t set for the nic only.
>>>>
>>>> Kambiz, just to quickly test it, can you set missing gateway for the
>>>>nic,
>>>> and stop/start the VR? And see if the start is completed normally,
>>>>just
>>>>to
>>>> find out if the missing gateway was the reason of the communication
>>>>failure
>>>>
>>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>>>>mode, state, removed from networks where id = 205;
>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>--
>>>>>+-
>>>>>------------+------+-----------+---------+
>>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>>|
>>>>>gateway | mode | state | removed |
>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>--
>>>>>+-
>>>>>------------+------+-----------+---------+
>>>>>| 205 | default | Guest | Vlan |
>>>>>10.124.99.0/24
>>>>>|
>>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>>+-----+---------+--------------+-----------------------+--------------
>>>>>--
>>>>>+-
>>>>>------------+------+-----------+---------+
>>>>>
>>>>>Cheers
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>>>>looks
>>>>>> like they’ve got programmed to correct devices on the backend per
>>>>>>your
>>>>>> prev email?
>>>>>>
>>>>>> iface eth0 inet static
>>>>>> address 169.254.1.59
>>>>>> netmask 255.255.0.0
>>>>>>
>>>>>> iface eth1 inet static
>>>>>> address 10.193.17.1
>>>>>> Netmask
>>>>>>
>>>>>> iface eth2 inet static
>>>>>> address 10.193.17.190
>>>>>> netmask 255.255.255.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> I can see that only one parameter is missing from the start command,
>>>>>>the
>>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>>> From the command/DB, I see that the gateway is missing in the nics
>>>>>>table
>>>>>> for the network 205? Can you check gateway information in the
>>>>>>networks
>>>>>> table for the id=205
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>select
>>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_nam
>>>>>>>e
>>>>>>>from nics where instance_id=19;
>>>>>>>
>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>------------+--------------------------+
>>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>>removed
>>>>>>>| network_id | reserver_name |
>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>------------+--------------------------+
>>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated |
>>>>>>>NULL
>>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>>| 30 | NULL | NULL | NULL | Allocated |
>>>>>>>NULL
>>>>>>>| 202 | ControlNetworkGuru |
>>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated |
>>>>>>>NULL
>>>>>>>| 200 | PublicNetworkGuru |
>>>>>>>+----+---------------+---------------+-------------+-----------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>------------+--------------------------+
>>>>>>>
>>>>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>>>>two nics with deviceId 0?
>>>>>>>
>>>>>>>"nics":[
>>>>>>> {"deviceId":2,
>>>>>>> "networkRateMbps":200,
>>>>>>> "defaultNic":true,
>>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>>> "ip":"10.193.17.190",
>>>>>>> "netmask":"255.255.255.0",
>>>>>>> "gateway":"10.193.17.1",
>>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>>> "dns1":"10.193.17.1",
>>>>>>> "broadcastType":"Vlan",
>>>>>>> "type":"Public",
>>>>>>> "broadcastUri":"vlan://untagged",
>>>>>>> "isolationUri":"vlan://untagged",
>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>> "name":"cloudbr0"},
>>>>>>> {"deviceId":0,
>>>>>>> "networkRateMbps":200,
>>>>>>> "defaultNic":false,
>>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>>> "ip":"10.124.99.1",
>>>>>>> "netmask":"255.255.255.0",
>>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>>> "dns1":"10.193.17.1",
>>>>>>> "broadcastType":"Vlan",
>>>>>>> "type":"Guest",
>>>>>>> "broadcastUri":"vlan://3949",
>>>>>>> "isolationUri":"vlan://3949",
>>>>>>> "isSecurityGroupEnabled":false,
>>>>>>> "name":"cloudbr1"},
>>>>>>> {"deviceId":0,
>>>>>>> "networkRateMbps":-1,
>>>>>>> "defaultNic":false,
>>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>>> "ip":"169.254.1.59",
>>>>>>> "netmask":"255.255.0.0",
>>>>>>> "gateway":"169.254.0.1",
>>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>>> "broadcastType":"LinkLocal",
>>>>>>> "type":"Control",
>>>>>>> "isSecurityGroupEnabled":false}
>>>>>>>]
>>>>>>>
>>>>>>>Thanks
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>>>>control
>>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>>command,
>>>>>>>> eth1 is not control, its either public or guest
>>>>>>>>
>>>>>>>> eth0: - control
>>>>>>>>
>>>>>>>> iface eth0 inet static
>>>>>>>> address 169.254.1.59
>>>>>>>> netmask 255.255.0.0
>>>>>>>>
>>>>>>>> eth1:
>>>>>>>>
>>>>>>>> iface eth1 inet static
>>>>>>>> address 10.193.17.1
>>>>>>>> Netmask
>>>>>>>>
>>>>>>>> So you need to execute the mysql statements for the traffic type
>>>>>>>>of
>>>>>>>>VR
>>>>>>>>nic
>>>>>>>> eth1
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>>><Al...@citrix.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>>
>>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>>traffic_type=‘Control’;
>>>>>>>>>
>>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>>
>>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>>>>network_id=<control network id>
>>>>>>>>>
>>>>>>>>>-Alena.
>>>>>>>>>
>>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>>>>worked
>>>>>>>>>>correctly:
>>>>>>>>>>
>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre
>>>>>>>>>>.s
>>>>>>>>>>h
>>>>>>>>>>-l
>>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.
>>>>>>>>>>25
>>>>>>>>>>5.
>>>>>>>>>>0%
>>>>>>>>>>ga
>>>>>>>>>>t
>>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain
>>>>>>>>>>=c
>>>>>>>>>>s6
>>>>>>>>>>cl
>>>>>>>>>>ou
>>>>>>>>>>d
>>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.
>>>>>>>>>>25
>>>>>>>>>>5.
>>>>>>>>>>0.
>>>>>>>>>>0%
>>>>>>>>>>t
>>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>
>>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip
>>>>>>>>>>and
>>>>>>>>>>eth1mask.
>>>>>>>>>>
>>>>>>>>>>Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>>eliminated
>>>>>>>>>>> static nat rule programming for non-existing vm. Missing
>>>>>>>>>>>netmask
>>>>>>>>>>>on
>>>>>>>>>>>eth1
>>>>>>>>>>> doesn’t seem related to the problem (although we have to figure
>>>>>>>>>>>out
>>>>>>>>>>>why
>>>>>>>>>>> its missing), as the connection that fails, happening to link
>>>>>>>>>>>local
>>>>>>>>>>>169.x
>>>>>>>>>>> eth0 interface.
>>>>>>>>>>>
>>>>>>>>>>> Edison, can you please tell us how to debug link local
>>>>>>>>>>>connection
>>>>>>>>>>>failure,
>>>>>>>>>>> on KVM agent?
>>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>> Alena.
>>>>>>>>>>>
>>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>>>>there
>>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>>
>>>>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>>>>there
>>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>>
>>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>>netmask
>>>>>>>>>>>>of
>>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>>
>>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>>
>>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>>
>>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>>...
>>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>>
>>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>>> netmask
>>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>>
>>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>>agent.log
>>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>>
>>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrp
>>>>>>>>>>>>re
>>>>>>>>>>>>.s
>>>>>>>>>>>>h
>>>>>>>>>>>>-l
>>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.2
>>>>>>>>>>>>55
>>>>>>>>>>>>.2
>>>>>>>>>>>>55
>>>>>>>>>>>>.0
>>>>>>>>>>>>%
>>>>>>>>>>>>ga
>>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%dom
>>>>>>>>>>>>ai
>>>>>>>>>>>>n=
>>>>>>>>>>>>cs
>>>>>>>>>>>>6c
>>>>>>>>>>>>l
>>>>>>>>>>>>ou
>>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=2
>>>>>>>>>>>>55
>>>>>>>>>>>>.2
>>>>>>>>>>>>55
>>>>>>>>>>>>.0
>>>>>>>>>>>>.
>>>>>>>>>>>>0%
>>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>>culprit.
>>>>>>>>>>>>>If
>>>>>>>>>>>>>vm
>>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it
>>>>>>>>>>>>>from
>>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>>>>where
>>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi"
>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>>following
>>>>>>>>>>>>>>query
>>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>>>>where
>>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>>+-------+------------+-------------------+--------------+----
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network
>>>>>>>>>>>>>>>(205,
>>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is
>>>>>>>>>>>>>>>non-removed
>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat
>>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>>> referring to vm that is removed already. If you find any,
>>>>>>>>>>>>>>>let
>>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi"
>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null,
>>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router
>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar
>>>>>>>>>>>>>>>>problem
>>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>---------+--------------------------+------------+---------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because the
>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in
>>>>>>>>>>>>>>>>>VPC
>>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to
>>>>>>>>>>>>>>>>>revert
>>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of
>>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to
>>>>>>>>>>>>>>>>>>thank
>>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote
>>>>>>>>>>>>>>>>>>debugging, I
>>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.
>>>>>>>>>>>>>>>>>>1]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to
>>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is
>>>>>>>>>>>>>>>>>>not
>>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.
>>>>>>>>>>>>>>>>>>99
>>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a
>>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress
>>>>>>>>>>>>>>>>>>rule(s)
>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(Network
>>>>>>>>>>>>>>>>>>Mo
>>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>>>>>>23
>>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManage
>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:7
>>>>>>>>>>>>>>>>>>67
>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtu
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachi
>>>>>>>>>>>>>>>>>>ne
>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.ja
>>>>>>>>>>>>>>>>>>va
>>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(
>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElem
>>>>>>>>>>>>>>>>>>en
>>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Net
>>>>>>>>>>>>>>>>>>wo
>>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(Network
>>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerIm
>>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProce
>>>>>>>>>>>>>>>>>>ss
>>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.j
>>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRoute
>>>>>>>>>>>>>>>>>>rC
>>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>| id | uuid |
>>>>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway |
>>>>>>>>>>>>>>>>>>ip_type |
>>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>>> | removed | ip6_gateway |
>>>>>>>>>>>>>>>>>>ip6_cidr |
>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |
>>>>>>>>>>>>>>>>>>1 |
>>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |
>>>>>>>>>>>>>>>>>>2 |
>>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved |
>>>>>>>>>>>>>>>>>>Start
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |
>>>>>>>>>>>>>>>>>>4
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |
>>>>>>>>>>>>>>>>>>7
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |
>>>>>>>>>>>>>>>>>>18
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |
>>>>>>>>>>>>>>>>>>19
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL
>>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>>+----+--------------------------------------+------------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+--------
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+--------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------------------------------+-----------+----------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>----------+-------------+-------------+------------------
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
I executed
> update nics set device_id = 1 where id = 29;
After restarting the router, the interfaces file now looks like this:
root@r-19-VM:~# cat /etc/network/interfaces
auto lo eth0 eth1 eth2
iface lo inet loopback
iface eth0 inet static
address 169.254.3.155
netmask 255.255.0.0
iface eth1 inet static
address 10.124.99.1
netmask 255.255.255.0
iface eth2 inet static
address 10.193.17.190
netmask 255.255.255.0
ifconfig shows this:
root@r-19-VM:~# ifconfig
eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:9b
inet addr:169.254.3.155 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::c00:a9ff:fefe:39b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:61 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3930 (3.8 KiB) TX bytes:730 (730.0 B)
eth1 Link encap:Ethernet HWaddr 02:00:2a:43:00:0d
inet addr:10.124.99.1 Bcast:10.124.99.255 Mask:255.255.255.0
inet6 addr: fe80::2aff:fe43:d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:936 (936.0 B) TX bytes:318 (318.0 B)
eth2 Link encap:Ethernet HWaddr 06:7e:fe:00:00:bf
inet addr:10.193.17.190 Bcast:10.193.17.255 Mask:255.255.255.0
inet6 addr: fe80::47e:feff:fe00:bf/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:846 (846.0 B) TX bytes:696 (696.0 B)
>From inside the VM, I can ping the gateways of the public, guest and
control network:
root@r-19-VM:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.124.99.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.193.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 10.193.17.1 0.0.0.0 UG 0 0 0 eth2
root@r-19-VM:~# ping 10.193.17.1
PING 10.193.17.1 (10.193.17.1): 56 data bytes
64 bytes from 10.193.17.1: icmp_seq=0 ttl=64 time=1.194 ms
64 bytes from 10.193.17.1: icmp_seq=1 ttl=64 time=0.329 ms
root@r-19-VM:~# ping 10.124.99.1
PING 10.124.99.1 (10.124.99.1): 56 data bytes
64 bytes from 10.124.99.1: icmp_seq=0 ttl=64 time=0.128 ms
root@r-19-VM:/etc/init.d# ping 169.254.0.1
PING 169.254.0.1 (169.254.0.1): 56 data bytes
64 bytes from 169.254.0.1: icmp_seq=0 ttl=64 time=0.292 ms
And from outside, I can ping the different IPs of the router.
But what is strange, is that in agent.log, I still find
Ping command port, 169.254.3.155:3922
Trying to connect to 169.254.3.155
Could not connect to 169.254.3.155
And when I check on the router, the ssh daemon only listens on the guest
network interface:
# netstat -na | grep 3922
tcp 0 0 10.124.99.1:3922 0.0.0.0:* LISTEN
So, the connection attempt to 169.254.3.155:3922 fails:
telnet 169.254.3.155 3922
Trying 169.254.3.155...
Is that the normal situation?
> Also after which point you started experiencing all this problems? Did
> you upgrade to new CS version? Or does it fail for any specific network?
No, I didn't upgrade to a new CS, I just stopped and started the
management-server.
Thanks
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Kambiz, did you check the device id in nics table for the vm? If it has 2
> 0s, change one of them to the correct value and restart. If the testing
> completes fine, we have a proof that its related to the device id mix up.
> If not, there is gotta be something else, most likely misconfigured on KVM
> stuff.
>
> Also after which point you started experiencing all this problems? Did
> you upgrade to new CS version? Or does it fail for any specific network?
>
> -Alena.
>
> On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>I updated nics.gateway for that network, but the VM still shows the same
>>behaviour.
>>
>>If one compares interfaces:
>>
>>root@r-19-VM:~# cat /etc/network/interfaces
>>auto lo eth0 eth1 eth2
>>iface lo inet loopback
>>
>>iface eth0 inet static
>> address 169.254.1.242
>> netmask 255.255.0.0
>>iface eth1 inet static
>> address 10.193.17.1
>> netmask
>>iface eth2 inet static
>> address 10.193.17.190
>> netmask 255.255.255.0
>>
>>and the nics entry in the management-server.log (cf. below), one can see
>>that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
>>correctly set up with IP 10.193.17.190.
>>
>>The remaining nic is eth1 which corresponds to the first nic with
>>deviceId 0 and according to the nics entry should have IP 10.124.99.1,
>>but there is no iface entry with that IP, but eth1 has the gateway
>>address of the public nic as its IP address.
>>
>>Could the problem have something to do with the duplicate deviceId 0?
>>
>>Thanks
>>
>>
>>Kambiz
>>
>>{ "nics":
>> [{"deviceId":2,
>> "networkRateMbps":200,
>> "defaultNic":true,
>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>> "ip":"10.193.17.190",
>> "netmask":"255.255.255.0",
>> "gateway":"10.193.17.1",
>> "mac":"06:7e:fe:00:00:bf",
>> "dns1":"10.193.17.1",
>> "broadcastType":"Vlan",
>> "type":"Public",
>> "broadcastUri":"vlan://untagged",
>> "isolationUri":"vlan://untagged",
>> "isSecurityGroupEnabled":false,
>> "name":"cloudbr0"},
>> {"deviceId":0,
>> "networkRateMbps":200,
>> "defaultNic":false,
>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>> "ip":"10.124.99.1",
>> "netmask":"255.255.255.0",
>> "gateway":"10.124.99.1",
>> "mac":"02:00:2a:43:00:0d",
>> "dns1":"10.193.17.1",
>> "broadcastType":"Vlan",
>> "type":"Guest",
>> "broadcastUri":"vlan://3925",
>> "isolationUri":"vlan://3925",
>> "isSecurityGroupEnabled":false,
>> "name":"cloudbr1"},
>> {"deviceId":0,
>> "networkRateMbps":-1,
>> "defaultNic":false,
>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>> "ip":"169.254.1.242",
>> "netmask":"255.255.0.0",
>> "gateway":"169.254.0.1",
>> "mac":"0e:00:a9:fe:01:f2",
>> "broadcastType":"LinkLocal",
>> "type":"Control",
>> "isSecurityGroupEnabled":false}
>> ]
>>}
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> So the gateway wasn’t set for the nic only.
>>>
>>> Kambiz, just to quickly test it, can you set missing gateway for the
>>>nic,
>>> and stop/start the VR? And see if the start is completed normally, just
>>>to
>>> find out if the missing gateway was the reason of the communication
>>>failure
>>>
>>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>>>mode, state, removed from networks where id = 205;
>>>>+-----+---------+--------------+-----------------------+----------------
>>>>+-
>>>>------------+------+-----------+---------+
>>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>>|
>>>>gateway | mode | state | removed |
>>>>+-----+---------+--------------+-----------------------+----------------
>>>>+-
>>>>------------+------+-----------+---------+
>>>>| 205 | default | Guest | Vlan | 10.124.99.0/24
>>>>|
>>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>>+-----+---------+--------------+-----------------------+----------------
>>>>+-
>>>>------------+------+-----------+---------+
>>>>
>>>>Cheers
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>>>looks
>>>>> like they’ve got programmed to correct devices on the backend per your
>>>>> prev email?
>>>>>
>>>>> iface eth0 inet static
>>>>> address 169.254.1.59
>>>>> netmask 255.255.0.0
>>>>>
>>>>> iface eth1 inet static
>>>>> address 10.193.17.1
>>>>> Netmask
>>>>>
>>>>> iface eth2 inet static
>>>>> address 10.193.17.190
>>>>> netmask 255.255.255.0
>>>>>
>>>>>
>>>>>
>>>>> I can see that only one parameter is missing from the start command,
>>>>>the
>>>>> second nic (network id=205) doesn’t have the gateway.
>>>>> From the command/DB, I see that the gateway is missing in the nics
>>>>>table
>>>>> for the network 205? Can you check gateway information in the networks
>>>>> table for the id=205
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>select
>>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>>>>>>from nics where instance_id=19;
>>>>>>
>>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>>--
>>>>>>-+
>>>>>>------------+--------------------------+
>>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>>removed
>>>>>>| network_id | reserver_name |
>>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>>--
>>>>>>-+
>>>>>>------------+--------------------------+
>>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>>| 30 | NULL | NULL | NULL | Allocated | NULL
>>>>>>| 202 | ControlNetworkGuru |
>>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>>>>>>| 200 | PublicNetworkGuru |
>>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>>--
>>>>>>-+
>>>>>>------------+--------------------------+
>>>>>>
>>>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>>>two nics with deviceId 0?
>>>>>>
>>>>>>"nics":[
>>>>>> {"deviceId":2,
>>>>>> "networkRateMbps":200,
>>>>>> "defaultNic":true,
>>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>>> "ip":"10.193.17.190",
>>>>>> "netmask":"255.255.255.0",
>>>>>> "gateway":"10.193.17.1",
>>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>>> "dns1":"10.193.17.1",
>>>>>> "broadcastType":"Vlan",
>>>>>> "type":"Public",
>>>>>> "broadcastUri":"vlan://untagged",
>>>>>> "isolationUri":"vlan://untagged",
>>>>>> "isSecurityGroupEnabled":false,
>>>>>> "name":"cloudbr0"},
>>>>>> {"deviceId":0,
>>>>>> "networkRateMbps":200,
>>>>>> "defaultNic":false,
>>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>>> "ip":"10.124.99.1",
>>>>>> "netmask":"255.255.255.0",
>>>>>> "mac":"02:00:2a:43:00:0d",
>>>>>> "dns1":"10.193.17.1",
>>>>>> "broadcastType":"Vlan",
>>>>>> "type":"Guest",
>>>>>> "broadcastUri":"vlan://3949",
>>>>>> "isolationUri":"vlan://3949",
>>>>>> "isSecurityGroupEnabled":false,
>>>>>> "name":"cloudbr1"},
>>>>>> {"deviceId":0,
>>>>>> "networkRateMbps":-1,
>>>>>> "defaultNic":false,
>>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>>> "ip":"169.254.1.59",
>>>>>> "netmask":"255.255.0.0",
>>>>>> "gateway":"169.254.0.1",
>>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>>> "broadcastType":"LinkLocal",
>>>>>> "type":"Control",
>>>>>> "isSecurityGroupEnabled":false}
>>>>>>]
>>>>>>
>>>>>>Thanks
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>>>control
>>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>>command,
>>>>>>> eth1 is not control, its either public or guest
>>>>>>>
>>>>>>> eth0: - control
>>>>>>>
>>>>>>> iface eth0 inet static
>>>>>>> address 169.254.1.59
>>>>>>> netmask 255.255.0.0
>>>>>>>
>>>>>>> eth1:
>>>>>>>
>>>>>>> iface eth1 inet static
>>>>>>> address 10.193.17.1
>>>>>>> Netmask
>>>>>>>
>>>>>>> So you need to execute the mysql statements for the traffic type of
>>>>>>>VR
>>>>>>>nic
>>>>>>> eth1
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>>><Al...@citrix.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>Kambiz, can you please check the following:
>>>>>>>>
>>>>>>>>
>>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>>
>>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>>traffic_type=‘Control’;
>>>>>>>>
>>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>>
>>>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>>>network_id=<control network id>
>>>>>>>>
>>>>>>>>-Alena.
>>>>>>>>
>>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>>>worked
>>>>>>>>>correctly:
>>>>>>>>>
>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.s
>>>>>>>>>h
>>>>>>>>>-l
>>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.25
>>>>>>>>>5.
>>>>>>>>>0%
>>>>>>>>>ga
>>>>>>>>>t
>>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=c
>>>>>>>>>s6
>>>>>>>>>cl
>>>>>>>>>ou
>>>>>>>>>d
>>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.25
>>>>>>>>>5.
>>>>>>>>>0.
>>>>>>>>>0%
>>>>>>>>>t
>>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>
>>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>>>>>>eth1mask.
>>>>>>>>>
>>>>>>>>>Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>>eliminated
>>>>>>>>>> static nat rule programming for non-existing vm. Missing netmask
>>>>>>>>>>on
>>>>>>>>>>eth1
>>>>>>>>>> doesn’t seem related to the problem (although we have to figure
>>>>>>>>>>out
>>>>>>>>>>why
>>>>>>>>>> its missing), as the connection that fails, happening to link
>>>>>>>>>>local
>>>>>>>>>>169.x
>>>>>>>>>> eth0 interface.
>>>>>>>>>>
>>>>>>>>>> Edison, can you please tell us how to debug link local connection
>>>>>>>>>>failure,
>>>>>>>>>> on KVM agent?
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>> Alena.
>>>>>>>>>>
>>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi,
>>>>>>>>>>>
>>>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>>>there
>>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>>
>>>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>>>there
>>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>>
>>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>>netmask
>>>>>>>>>>>of
>>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>>
>>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>>Escape character is ^]
>>>>>>>>>>>
>>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>>
>>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>>...
>>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>>
>>>>>>>>>>>iface eth0 inet static
>>>>>>>>>>> address 169.254.1.59
>>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>>iface eth1 inet static
>>>>>>>>>>> address 10.193.17.1
>>>>>>>>>>> netmask
>>>>>>>>>>>iface eth2 inet static
>>>>>>>>>>> address 10.193.17.190
>>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>>
>>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>>agent.log
>>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>>
>>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre
>>>>>>>>>>>.s
>>>>>>>>>>>h
>>>>>>>>>>>-l
>>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255
>>>>>>>>>>>.2
>>>>>>>>>>>55
>>>>>>>>>>>.0
>>>>>>>>>>>%
>>>>>>>>>>>ga
>>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domai
>>>>>>>>>>>n=
>>>>>>>>>>>cs
>>>>>>>>>>>6c
>>>>>>>>>>>l
>>>>>>>>>>>ou
>>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255
>>>>>>>>>>>.2
>>>>>>>>>>>55
>>>>>>>>>>>.0
>>>>>>>>>>>.
>>>>>>>>>>>0%
>>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>>
>>>>>>>>>>>Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>>culprit.
>>>>>>>>>>>>If
>>>>>>>>>>>>vm
>>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it
>>>>>>>>>>>>from
>>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>>
>>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>>>where
>>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>
>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>>following
>>>>>>>>>>>>>query
>>>>>>>>>>>>>correctly:
>>>>>>>>>>>>>
>>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>>>where
>>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>>nic_removed
>>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>>
>>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>
>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>>id,
>>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For each vmId/networkId combo, check if there is non-removed
>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat
>>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>>reference
>>>>>>>>>>>>>> referring to vm that is removed already. If you find any, let
>>>>>>>>>>>>>>me
>>>>>>>>>>>>>>know
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null,
>>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The error happens not because Ip is null, but because the
>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC
>>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of
>>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to thank
>>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part
>>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s)
>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkMo
>>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:23
>>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerI
>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.j
>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerI
>>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767
>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtual
>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachine
>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java
>>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Vi
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElemen
>>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Netwo
>>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.jav
>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkMa
>>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.jav
>>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterC
>>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL |
>>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Kambiz, did you check the device id in nics table for the vm? If it has 2
0s, change one of them to the correct value and restart. If the testing
completes fine, we have a proof that its related to the device id mix up.
If not, there is gotta be something else, most likely misconfigured on KVM
stuff.
Also after which point you started experiencing all this problems? Did
you upgrade to new CS version? Or does it fail for any specific network?
-Alena.
On 3/25/14, 2:11 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>I updated nics.gateway for that network, but the VM still shows the same
>behaviour.
>
>If one compares interfaces:
>
>root@r-19-VM:~# cat /etc/network/interfaces
>auto lo eth0 eth1 eth2
>iface lo inet loopback
>
>iface eth0 inet static
> address 169.254.1.242
> netmask 255.255.0.0
>iface eth1 inet static
> address 10.193.17.1
> netmask
>iface eth2 inet static
> address 10.193.17.190
> netmask 255.255.255.0
>
>and the nics entry in the management-server.log (cf. below), one can see
>that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
>correctly set up with IP 10.193.17.190.
>
>The remaining nic is eth1 which corresponds to the first nic with
>deviceId 0 and according to the nics entry should have IP 10.124.99.1,
>but there is no iface entry with that IP, but eth1 has the gateway
>address of the public nic as its IP address.
>
>Could the problem have something to do with the duplicate deviceId 0?
>
>Thanks
>
>
>Kambiz
>
>{ "nics":
> [{"deviceId":2,
> "networkRateMbps":200,
> "defaultNic":true,
> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
> "ip":"10.193.17.190",
> "netmask":"255.255.255.0",
> "gateway":"10.193.17.1",
> "mac":"06:7e:fe:00:00:bf",
> "dns1":"10.193.17.1",
> "broadcastType":"Vlan",
> "type":"Public",
> "broadcastUri":"vlan://untagged",
> "isolationUri":"vlan://untagged",
> "isSecurityGroupEnabled":false,
> "name":"cloudbr0"},
> {"deviceId":0,
> "networkRateMbps":200,
> "defaultNic":false,
> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
> "ip":"10.124.99.1",
> "netmask":"255.255.255.0",
> "gateway":"10.124.99.1",
> "mac":"02:00:2a:43:00:0d",
> "dns1":"10.193.17.1",
> "broadcastType":"Vlan",
> "type":"Guest",
> "broadcastUri":"vlan://3925",
> "isolationUri":"vlan://3925",
> "isSecurityGroupEnabled":false,
> "name":"cloudbr1"},
> {"deviceId":0,
> "networkRateMbps":-1,
> "defaultNic":false,
> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
> "ip":"169.254.1.242",
> "netmask":"255.255.0.0",
> "gateway":"169.254.0.1",
> "mac":"0e:00:a9:fe:01:f2",
> "broadcastType":"LinkLocal",
> "type":"Control",
> "isSecurityGroupEnabled":false}
> ]
>}
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> So the gateway wasn’t set for the nic only.
>>
>> Kambiz, just to quickly test it, can you set missing gateway for the
>>nic,
>> and stop/start the VR? And see if the start is completed normally, just
>>to
>> find out if the missing gateway was the reason of the communication
>>failure
>>
>> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>>mode, state, removed from networks where id = 205;
>>>+-----+---------+--------------+-----------------------+----------------
>>>+-
>>>------------+------+-----------+---------+
>>>| id | name | traffic_type | broadcast_domain_type | cidr
>>>|
>>>gateway | mode | state | removed |
>>>+-----+---------+--------------+-----------------------+----------------
>>>+-
>>>------------+------+-----------+---------+
>>>| 205 | default | Guest | Vlan | 10.124.99.0/24
>>>|
>>>10.124.99.1 | Dhcp | Allocated | NULL |
>>>+-----+---------+--------------+-----------------------+----------------
>>>+-
>>>------------+------+-----------+---------+
>>>
>>>Cheers
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>>looks
>>>> like they’ve got programmed to correct devices on the backend per your
>>>> prev email?
>>>>
>>>> iface eth0 inet static
>>>> address 169.254.1.59
>>>> netmask 255.255.0.0
>>>>
>>>> iface eth1 inet static
>>>> address 10.193.17.1
>>>> Netmask
>>>>
>>>> iface eth2 inet static
>>>> address 10.193.17.190
>>>> netmask 255.255.255.0
>>>>
>>>>
>>>>
>>>> I can see that only one parameter is missing from the start command,
>>>>the
>>>> second nic (network id=205) doesn’t have the gateway.
>>>> From the command/DB, I see that the gateway is missing in the nics
>>>>table
>>>> for the network 205? Can you check gateway information in the networks
>>>> table for the id=205
>>>>
>>>>
>>>>
>>>>
>>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>select
>>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>>>>>from nics where instance_id=19;
>>>>>
>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>--
>>>>>-+
>>>>>------------+--------------------------+
>>>>>| id | ip4_address | netmask | gateway | state |
>>>>>removed
>>>>>| network_id | reserver_name |
>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>--
>>>>>-+
>>>>>------------+--------------------------+
>>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>>>>>| 205 | ExternalGuestNetworkGuru |
>>>>>| 30 | NULL | NULL | NULL | Allocated | NULL
>>>>>| 202 | ControlNetworkGuru |
>>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>>>>>| 200 | PublicNetworkGuru |
>>>>>+----+---------------+---------------+-------------+-----------+------
>>>>>--
>>>>>-+
>>>>>------------+--------------------------+
>>>>>
>>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>>two nics with deviceId 0?
>>>>>
>>>>>"nics":[
>>>>> {"deviceId":2,
>>>>> "networkRateMbps":200,
>>>>> "defaultNic":true,
>>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>>> "ip":"10.193.17.190",
>>>>> "netmask":"255.255.255.0",
>>>>> "gateway":"10.193.17.1",
>>>>> "mac":"06:7e:fe:00:00:bf",
>>>>> "dns1":"10.193.17.1",
>>>>> "broadcastType":"Vlan",
>>>>> "type":"Public",
>>>>> "broadcastUri":"vlan://untagged",
>>>>> "isolationUri":"vlan://untagged",
>>>>> "isSecurityGroupEnabled":false,
>>>>> "name":"cloudbr0"},
>>>>> {"deviceId":0,
>>>>> "networkRateMbps":200,
>>>>> "defaultNic":false,
>>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>>> "ip":"10.124.99.1",
>>>>> "netmask":"255.255.255.0",
>>>>> "mac":"02:00:2a:43:00:0d",
>>>>> "dns1":"10.193.17.1",
>>>>> "broadcastType":"Vlan",
>>>>> "type":"Guest",
>>>>> "broadcastUri":"vlan://3949",
>>>>> "isolationUri":"vlan://3949",
>>>>> "isSecurityGroupEnabled":false,
>>>>> "name":"cloudbr1"},
>>>>> {"deviceId":0,
>>>>> "networkRateMbps":-1,
>>>>> "defaultNic":false,
>>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>>> "ip":"169.254.1.59",
>>>>> "netmask":"255.255.0.0",
>>>>> "gateway":"169.254.0.1",
>>>>> "mac":"0e:00:a9:fe:01:3b",
>>>>> "broadcastType":"LinkLocal",
>>>>> "type":"Control",
>>>>> "isSecurityGroupEnabled":false}
>>>>>]
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>>control
>>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>>command,
>>>>>> eth1 is not control, its either public or guest
>>>>>>
>>>>>> eth0: - control
>>>>>>
>>>>>> iface eth0 inet static
>>>>>> address 169.254.1.59
>>>>>> netmask 255.255.0.0
>>>>>>
>>>>>> eth1:
>>>>>>
>>>>>> iface eth1 inet static
>>>>>> address 10.193.17.1
>>>>>> Netmask
>>>>>>
>>>>>> So you need to execute the mysql statements for the traffic type of
>>>>>>VR
>>>>>>nic
>>>>>> eth1
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>>><Al...@citrix.com>
>>>>>> wrote:
>>>>>>
>>>>>>>Kambiz, can you please check the following:
>>>>>>>
>>>>>>>
>>>>>>>1) Check if the gateway is set on control network:
>>>>>>>
>>>>>>>mysql> select gateway, cidr from networks where
>>>>>>>traffic_type=‘Control’;
>>>>>>>
>>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>>
>>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>>network_id=<control network id>
>>>>>>>
>>>>>>>-Alena.
>>>>>>>
>>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>>worked
>>>>>>>>correctly:
>>>>>>>>
>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.s
>>>>>>>>h
>>>>>>>>-l
>>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.25
>>>>>>>>5.
>>>>>>>>0%
>>>>>>>>ga
>>>>>>>>t
>>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=c
>>>>>>>>s6
>>>>>>>>cl
>>>>>>>>ou
>>>>>>>>d
>>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.25
>>>>>>>>5.
>>>>>>>>0.
>>>>>>>>0%
>>>>>>>>t
>>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>
>>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>>>>>eth1mask.
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>>eliminated
>>>>>>>>> static nat rule programming for non-existing vm. Missing netmask
>>>>>>>>>on
>>>>>>>>>eth1
>>>>>>>>> doesn’t seem related to the problem (although we have to figure
>>>>>>>>>out
>>>>>>>>>why
>>>>>>>>> its missing), as the connection that fails, happening to link
>>>>>>>>>local
>>>>>>>>>169.x
>>>>>>>>> eth0 interface.
>>>>>>>>>
>>>>>>>>> Edison, can you please tell us how to debug link local connection
>>>>>>>>>failure,
>>>>>>>>> on KVM agent?
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> Alena.
>>>>>>>>>
>>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>>there
>>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>>
>>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>>there
>>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>>
>>>>>>>>>>Then I logged into the virtual router to find out that the
>>>>>>>>>>netmask
>>>>>>>>>>of
>>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>>
>>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>>Escape character is ^]
>>>>>>>>>>
>>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>>
>>>>>>>>>>r-19-VM login: root
>>>>>>>>>>...
>>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>>iface lo inet loopback
>>>>>>>>>>
>>>>>>>>>>iface eth0 inet static
>>>>>>>>>> address 169.254.1.59
>>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>>iface eth1 inet static
>>>>>>>>>> address 10.193.17.1
>>>>>>>>>> netmask
>>>>>>>>>>iface eth2 inet static
>>>>>>>>>> address 10.193.17.190
>>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>>
>>>>>>>>>>I don't know if it is relevant, but this is the line from
>>>>>>>>>>agent.log
>>>>>>>>>>where the parameters are visible:
>>>>>>>>>>
>>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre
>>>>>>>>>>.s
>>>>>>>>>>h
>>>>>>>>>>-l
>>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255
>>>>>>>>>>.2
>>>>>>>>>>55
>>>>>>>>>>.0
>>>>>>>>>>%
>>>>>>>>>>ga
>>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domai
>>>>>>>>>>n=
>>>>>>>>>>cs
>>>>>>>>>>6c
>>>>>>>>>>l
>>>>>>>>>>ou
>>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255
>>>>>>>>>>.2
>>>>>>>>>>55
>>>>>>>>>>.0
>>>>>>>>>>.
>>>>>>>>>>0%
>>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Any hint is appreciated.
>>>>>>>>>>
>>>>>>>>>>Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the
>>>>>>>>>>>culprit.
>>>>>>>>>>>If
>>>>>>>>>>>vm
>>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it
>>>>>>>>>>>from
>>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>>
>>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>>
>>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>>where
>>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>I hope I have understood what you wrote and created the
>>>>>>>>>>>>following
>>>>>>>>>>>>query
>>>>>>>>>>>>correctly:
>>>>>>>>>>>>
>>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>>where
>>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>>nic_removed
>>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>>+-------+------------+-------------------+--------------+------
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>>
>>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>>
>>>>>>>>>>>>Thank you
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>
>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>>>>>right?)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those
>>>>>>>>>>>>>rules:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>>(Select
>>>>>>>>>>>>>id,
>>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>>
>>>>>>>>>>>>> For each vmId/networkId combo, check if there is non-removed
>>>>>>>>>>>>>nic
>>>>>>>>>>>>>and
>>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat
>>>>>>>>>>>>>ip/vm
>>>>>>>>>>>>>reference
>>>>>>>>>>>>> referring to vm that is removed already. If you find any, let
>>>>>>>>>>>>>me
>>>>>>>>>>>>>know
>>>>>>>>>>>>>and
>>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null,
>>>>>>>>>>>>>>but
>>>>>>>>>>>>>>I
>>>>>>>>>>>>>>removed
>>>>>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I
>>>>>>>>>>>>>>reverted
>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>>router
>>>>>>>>>>>>>>which
>>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>>after
>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>first
>>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as
>>>>>>>>>>>>>>instance_id,
>>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>>name
>>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>| 30 | NULL | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>>+----+---------------------+---------------+---------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>---------+--------------------------+------------+-----------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The error happens not because Ip is null, but because the
>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC
>>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of
>>>>>>>>>>>>>>>your
>>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi"
>>>>>>>>>>>>>>><da...@m-creations.com>
>>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>as this is my first post to the list, I would like to thank
>>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers
>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id
>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id
>>>>>>>>>>>>>>>>202
>>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99
>>>>>>>>>>>>>>>>]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1
>>>>>>>>>>>>>>>>eth0mask=255.255.255.0
>>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part
>>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port
>>>>>>>>>>>>>>>>forwarding,
>>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s)
>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkMo
>>>>>>>>>>>>>>>>de
>>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:23
>>>>>>>>>>>>>>>>46
>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerI
>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.j
>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerI
>>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767
>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.f
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(Virtual
>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachine
>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.d
>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java
>>>>>>>>>>>>>>>>:1
>>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Vi
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElemen
>>>>>>>>>>>>>>>>ts
>>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Netwo
>>>>>>>>>>>>>>>>rk
>>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.jav
>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkMa
>>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl
>>>>>>>>>>>>>>>>.s
>>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcess
>>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.jav
>>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterC
>>>>>>>>>>>>>>>>md
>>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 |
>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 |
>>>>>>>>>>>>>>>>2014-03-17
>>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 |
>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 |
>>>>>>>>>>>>>>>>2014-03-18
>>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL |
>>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+------------+--------+--------------+----------+----------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------------------------------+-----------+------------
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------
>>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
I updated nics.gateway for that network, but the VM still shows the same
behaviour.
If one compares interfaces:
root@r-19-VM:~# cat /etc/network/interfaces
auto lo eth0 eth1 eth2
iface lo inet loopback
iface eth0 inet static
address 169.254.1.242
netmask 255.255.0.0
iface eth1 inet static
address 10.193.17.1
netmask
iface eth2 inet static
address 10.193.17.190
netmask 255.255.255.0
and the nics entry in the management-server.log (cf. below), one can see
that eth0 is the second nic with deviceId 0 of type 'Control', eth2 is
correctly set up with IP 10.193.17.190.
The remaining nic is eth1 which corresponds to the first nic with
deviceId 0 and according to the nics entry should have IP 10.124.99.1,
but there is no iface entry with that IP, but eth1 has the gateway
address of the public nic as its IP address.
Could the problem have something to do with the duplicate deviceId 0?
Thanks
Kambiz
{ "nics":
[{"deviceId":2,
"networkRateMbps":200,
"defaultNic":true,
"uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
"ip":"10.193.17.190",
"netmask":"255.255.255.0",
"gateway":"10.193.17.1",
"mac":"06:7e:fe:00:00:bf",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Public",
"broadcastUri":"vlan://untagged",
"isolationUri":"vlan://untagged",
"isSecurityGroupEnabled":false,
"name":"cloudbr0"},
{"deviceId":0,
"networkRateMbps":200,
"defaultNic":false,
"uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
"ip":"10.124.99.1",
"netmask":"255.255.255.0",
"gateway":"10.124.99.1",
"mac":"02:00:2a:43:00:0d",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Guest",
"broadcastUri":"vlan://3925",
"isolationUri":"vlan://3925",
"isSecurityGroupEnabled":false,
"name":"cloudbr1"},
{"deviceId":0,
"networkRateMbps":-1,
"defaultNic":false,
"uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
"ip":"169.254.1.242",
"netmask":"255.255.0.0",
"gateway":"169.254.0.1",
"mac":"0e:00:a9:fe:01:f2",
"broadcastType":"LinkLocal",
"type":"Control",
"isSecurityGroupEnabled":false}
]
}
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> So the gateway wasn’t set for the nic only.
>
> Kambiz, just to quickly test it, can you set missing gateway for the nic,
> and stop/start the VR? And see if the start is completed normally, just to
> find out if the missing gateway was the reason of the communication failure
>
> On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>>mode, state, removed from networks where id = 205;
>>+-----+---------+--------------+-----------------------+----------------+-
>>------------+------+-----------+---------+
>>| id | name | traffic_type | broadcast_domain_type | cidr |
>>gateway | mode | state | removed |
>>+-----+---------+--------------+-----------------------+----------------+-
>>------------+------+-----------+---------+
>>| 205 | default | Guest | Vlan | 10.124.99.0/24 |
>>10.124.99.1 | Dhcp | Allocated | NULL |
>>+-----+---------+--------------+-----------------------+----------------+-
>>------------+------+-----------+---------+
>>
>>Cheers
>>
>>
>>Kambiz
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>>looks
>>> like they’ve got programmed to correct devices on the backend per your
>>> prev email?
>>>
>>> iface eth0 inet static
>>> address 169.254.1.59
>>> netmask 255.255.0.0
>>>
>>> iface eth1 inet static
>>> address 10.193.17.1
>>> Netmask
>>>
>>> iface eth2 inet static
>>> address 10.193.17.190
>>> netmask 255.255.255.0
>>>
>>>
>>>
>>> I can see that only one parameter is missing from the start command, the
>>> second nic (network id=205) doesn’t have the gateway.
>>> From the command/DB, I see that the gateway is missing in the nics table
>>> for the network 205? Can you check gateway information in the networks
>>> table for the id=205
>>>
>>>
>>>
>>>
>>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>select
>>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>>>>from nics where instance_id=19;
>>>>
>>>>+----+---------------+---------------+-------------+-----------+--------
>>>>-+
>>>>------------+--------------------------+
>>>>| id | ip4_address | netmask | gateway | state | removed
>>>>| network_id | reserver_name |
>>>>+----+---------------+---------------+-------------+-----------+--------
>>>>-+
>>>>------------+--------------------------+
>>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>>>>| 205 | ExternalGuestNetworkGuru |
>>>>| 30 | NULL | NULL | NULL | Allocated | NULL
>>>>| 202 | ControlNetworkGuru |
>>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>>>>| 200 | PublicNetworkGuru |
>>>>+----+---------------+---------------+-------------+-----------+--------
>>>>-+
>>>>------------+--------------------------+
>>>>
>>>>and this is the nics element from the StartCmd. Is it normal to have
>>>>two nics with deviceId 0?
>>>>
>>>>"nics":[
>>>> {"deviceId":2,
>>>> "networkRateMbps":200,
>>>> "defaultNic":true,
>>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>>> "ip":"10.193.17.190",
>>>> "netmask":"255.255.255.0",
>>>> "gateway":"10.193.17.1",
>>>> "mac":"06:7e:fe:00:00:bf",
>>>> "dns1":"10.193.17.1",
>>>> "broadcastType":"Vlan",
>>>> "type":"Public",
>>>> "broadcastUri":"vlan://untagged",
>>>> "isolationUri":"vlan://untagged",
>>>> "isSecurityGroupEnabled":false,
>>>> "name":"cloudbr0"},
>>>> {"deviceId":0,
>>>> "networkRateMbps":200,
>>>> "defaultNic":false,
>>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>>> "ip":"10.124.99.1",
>>>> "netmask":"255.255.255.0",
>>>> "mac":"02:00:2a:43:00:0d",
>>>> "dns1":"10.193.17.1",
>>>> "broadcastType":"Vlan",
>>>> "type":"Guest",
>>>> "broadcastUri":"vlan://3949",
>>>> "isolationUri":"vlan://3949",
>>>> "isSecurityGroupEnabled":false,
>>>> "name":"cloudbr1"},
>>>> {"deviceId":0,
>>>> "networkRateMbps":-1,
>>>> "defaultNic":false,
>>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>>> "ip":"169.254.1.59",
>>>> "netmask":"255.255.0.0",
>>>> "gateway":"169.254.0.1",
>>>> "mac":"0e:00:a9:fe:01:3b",
>>>> "broadcastType":"LinkLocal",
>>>> "type":"Control",
>>>> "isSecurityGroupEnabled":false}
>>>>]
>>>>
>>>>Thanks
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>>control
>>>>> interface as it was in your old command. I’ve looked at the new
>>>>>command,
>>>>> eth1 is not control, its either public or guest
>>>>>
>>>>> eth0: - control
>>>>>
>>>>> iface eth0 inet static
>>>>> address 169.254.1.59
>>>>> netmask 255.255.0.0
>>>>>
>>>>> eth1:
>>>>>
>>>>> iface eth1 inet static
>>>>> address 10.193.17.1
>>>>> Netmask
>>>>>
>>>>> So you need to execute the mysql statements for the traffic type of VR
>>>>>nic
>>>>> eth1
>>>>>
>>>>> -Alena.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>>><Al...@citrix.com>
>>>>> wrote:
>>>>>
>>>>>>Kambiz, can you please check the following:
>>>>>>
>>>>>>
>>>>>>1) Check if the gateway is set on control network:
>>>>>>
>>>>>>mysql> select gateway, cidr from networks where
>>>>>>traffic_type=‘Control’;
>>>>>>
>>>>>>2) For router control nic, check if network/gateway are set.
>>>>>>
>>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>>network_id=<control network id>
>>>>>>
>>>>>>-Alena.
>>>>>>
>>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>I looked up the startup command of the old router instance which
>>>>>>>worked
>>>>>>>correctly:
>>>>>>>
>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh
>>>>>>>-l
>>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.
>>>>>>>0%
>>>>>>>ga
>>>>>>>t
>>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6
>>>>>>>cl
>>>>>>>ou
>>>>>>>d
>>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.
>>>>>>>0.
>>>>>>>0%
>>>>>>>t
>>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>
>>>>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>>>>eth1mask.
>>>>>>>
>>>>>>>Thanks
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>>eliminated
>>>>>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>>>>>eth1
>>>>>>>> doesn’t seem related to the problem (although we have to figure out
>>>>>>>>why
>>>>>>>> its missing), as the connection that fails, happening to link local
>>>>>>>>169.x
>>>>>>>> eth0 interface.
>>>>>>>>
>>>>>>>> Edison, can you please tell us how to debug link local connection
>>>>>>>>failure,
>>>>>>>> on KVM agent?
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>> Alena.
>>>>>>>>
>>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>>there
>>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>>
>>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>>there
>>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>>
>>>>>>>>>Then I logged into the virtual router to find out that the netmask
>>>>>>>>>of
>>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>>
>>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>>Connected to domain r-19-VM
>>>>>>>>>Escape character is ^]
>>>>>>>>>
>>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>>
>>>>>>>>>r-19-VM login: root
>>>>>>>>>...
>>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>>iface lo inet loopback
>>>>>>>>>
>>>>>>>>>iface eth0 inet static
>>>>>>>>> address 169.254.1.59
>>>>>>>>> netmask 255.255.0.0
>>>>>>>>>iface eth1 inet static
>>>>>>>>> address 10.193.17.1
>>>>>>>>> netmask
>>>>>>>>>iface eth2 inet static
>>>>>>>>> address 10.193.17.190
>>>>>>>>> netmask 255.255.255.0
>>>>>>>>>
>>>>>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>>>>>where the parameters are visible:
>>>>>>>>>
>>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.s
>>>>>>>>>h
>>>>>>>>>-l
>>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.2
>>>>>>>>>55
>>>>>>>>>.0
>>>>>>>>>%
>>>>>>>>>ga
>>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=
>>>>>>>>>cs
>>>>>>>>>6c
>>>>>>>>>l
>>>>>>>>>ou
>>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.2
>>>>>>>>>55
>>>>>>>>>.0
>>>>>>>>>.
>>>>>>>>>0%
>>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Any hint is appreciated.
>>>>>>>>>
>>>>>>>>>Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit.
>>>>>>>>>>If
>>>>>>>>>>vm
>>>>>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>>
>>>>>>>>>> 1) Save the db dump.
>>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>>
>>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>>where
>>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Let me know how it works.
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi,
>>>>>>>>>>>
>>>>>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>>>>>query
>>>>>>>>>>>correctly:
>>>>>>>>>>>
>>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>>from user_ip_address uip
>>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>>where
>>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>>nic_removed
>>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>>2014-03-18
>>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>>
>>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>>
>>>>>>>>>>>Thank you
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>>
>>>>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>>>>right?)
>>>>>>>>>>>>
>>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>>network_id=205;
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>>>>>
>>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>>(Select
>>>>>>>>>>>>id,
>>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>>
>>>>>>>>>>>> For each vmId/networkId combo, check if there is non-removed
>>>>>>>>>>>>nic
>>>>>>>>>>>>and
>>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>>>>>reference
>>>>>>>>>>>> referring to vm that is removed already. If you find any, let
>>>>>>>>>>>>me
>>>>>>>>>>>>know
>>>>>>>>>>>>and
>>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>>
>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>>
>>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>>
>>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but
>>>>>>>>>>>>>I
>>>>>>>>>>>>>removed
>>>>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>>>>created
>>>>>>>>>>>>>which also couldn't be started due to the same NPE. I reverted
>>>>>>>>>>>>>the
>>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>>
>>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>>router
>>>>>>>>>>>>>which
>>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>>after
>>>>>>>>>>>>>the
>>>>>>>>>>>>>first
>>>>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>>>>suggested
>>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>>procedure
>>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>>
>>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thank you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>
>>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>>>>>i.name,
>>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>>i.id
>>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>>>>gateway
>>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>>name
>>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>4 |
>>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>7 |
>>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>18 |
>>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>| 30 | NULL | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>>19 |
>>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>>
>>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The error happens not because Ip is null, but because the nic
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>certain
>>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC
>>>>>>>>>>>>>>nic
>>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>>>>back.
>>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of your
>>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>>of
>>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>>> table.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>as this is my first post to the list, I would like to thank
>>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL in
>>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202
>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202
>>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of
>>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding,
>>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkMode
>>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346
>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImp
>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.jav
>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImp
>>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMa
>>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMa
>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.d
>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1
>>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Virt
>>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElements
>>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Network
>>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor
>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:
>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkMana
>>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor
>>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:
>>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>created
>>>>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL |
>>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
So the gateway wasn’t set for the nic only.
Kambiz, just to quickly test it, can you set missing gateway for the nic,
and stop/start the VR? And see if the start is completed normally, just to
find out if the missing gateway was the reason of the communication failure
On 3/25/14, 1:31 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>select id, name, traffic_type, broadcast_domain_type, cidr, gateway,
>mode, state, removed from networks where id = 205;
>+-----+---------+--------------+-----------------------+----------------+-
>------------+------+-----------+---------+
>| id | name | traffic_type | broadcast_domain_type | cidr |
>gateway | mode | state | removed |
>+-----+---------+--------------+-----------------------+----------------+-
>------------+------+-----------+---------+
>| 205 | default | Guest | Vlan | 10.124.99.0/24 |
>10.124.99.1 | Dhcp | Allocated | NULL |
>+-----+---------+--------------+-----------------------+----------------+-
>------------+------+-----------+---------+
>
>Cheers
>
>
>Kambiz
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> No, it doesn’t seem right to me having 2 nics with device id 0. But
>>looks
>> like they’ve got programmed to correct devices on the backend per your
>> prev email?
>>
>> iface eth0 inet static
>> address 169.254.1.59
>> netmask 255.255.0.0
>>
>> iface eth1 inet static
>> address 10.193.17.1
>> Netmask
>>
>> iface eth2 inet static
>> address 10.193.17.190
>> netmask 255.255.255.0
>>
>>
>>
>> I can see that only one parameter is missing from the start command, the
>> second nic (network id=205) doesn’t have the gateway.
>> From the command/DB, I see that the gateway is missing in the nics table
>> for the network 205? Can you check gateway information in the networks
>> table for the id=205
>>
>>
>>
>>
>> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>select
>>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>>>from nics where instance_id=19;
>>>
>>>+----+---------------+---------------+-------------+-----------+--------
>>>-+
>>>------------+--------------------------+
>>>| id | ip4_address | netmask | gateway | state | removed
>>>| network_id | reserver_name |
>>>+----+---------------+---------------+-------------+-----------+--------
>>>-+
>>>------------+--------------------------+
>>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>>>| 205 | ExternalGuestNetworkGuru |
>>>| 30 | NULL | NULL | NULL | Allocated | NULL
>>>| 202 | ControlNetworkGuru |
>>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>>>| 200 | PublicNetworkGuru |
>>>+----+---------------+---------------+-------------+-----------+--------
>>>-+
>>>------------+--------------------------+
>>>
>>>and this is the nics element from the StartCmd. Is it normal to have
>>>two nics with deviceId 0?
>>>
>>>"nics":[
>>> {"deviceId":2,
>>> "networkRateMbps":200,
>>> "defaultNic":true,
>>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>>> "ip":"10.193.17.190",
>>> "netmask":"255.255.255.0",
>>> "gateway":"10.193.17.1",
>>> "mac":"06:7e:fe:00:00:bf",
>>> "dns1":"10.193.17.1",
>>> "broadcastType":"Vlan",
>>> "type":"Public",
>>> "broadcastUri":"vlan://untagged",
>>> "isolationUri":"vlan://untagged",
>>> "isSecurityGroupEnabled":false,
>>> "name":"cloudbr0"},
>>> {"deviceId":0,
>>> "networkRateMbps":200,
>>> "defaultNic":false,
>>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>>> "ip":"10.124.99.1",
>>> "netmask":"255.255.255.0",
>>> "mac":"02:00:2a:43:00:0d",
>>> "dns1":"10.193.17.1",
>>> "broadcastType":"Vlan",
>>> "type":"Guest",
>>> "broadcastUri":"vlan://3949",
>>> "isolationUri":"vlan://3949",
>>> "isSecurityGroupEnabled":false,
>>> "name":"cloudbr1"},
>>> {"deviceId":0,
>>> "networkRateMbps":-1,
>>> "defaultNic":false,
>>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>>> "ip":"169.254.1.59",
>>> "netmask":"255.255.0.0",
>>> "gateway":"169.254.0.1",
>>> "mac":"0e:00:a9:fe:01:3b",
>>> "broadcastType":"LinkLocal",
>>> "type":"Control",
>>> "isSecurityGroupEnabled":false}
>>>]
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>>control
>>>> interface as it was in your old command. I’ve looked at the new
>>>>command,
>>>> eth1 is not control, its either public or guest
>>>>
>>>> eth0: - control
>>>>
>>>> iface eth0 inet static
>>>> address 169.254.1.59
>>>> netmask 255.255.0.0
>>>>
>>>> eth1:
>>>>
>>>> iface eth1 inet static
>>>> address 10.193.17.1
>>>> Netmask
>>>>
>>>> So you need to execute the mysql statements for the traffic type of VR
>>>>nic
>>>> eth1
>>>>
>>>> -Alena.
>>>>
>>>>
>>>>
>>>>
>>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk"
>>>><Al...@citrix.com>
>>>> wrote:
>>>>
>>>>>Kambiz, can you please check the following:
>>>>>
>>>>>
>>>>>1) Check if the gateway is set on control network:
>>>>>
>>>>>mysql> select gateway, cidr from networks where
>>>>>traffic_type=‘Control’;
>>>>>
>>>>>2) For router control nic, check if network/gateway are set.
>>>>>
>>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>>network_id=<control network id>
>>>>>
>>>>>-Alena.
>>>>>
>>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>I looked up the startup command of the old router instance which
>>>>>>worked
>>>>>>correctly:
>>>>>>
>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh
>>>>>>-l
>>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.
>>>>>>0%
>>>>>>ga
>>>>>>t
>>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6
>>>>>>cl
>>>>>>ou
>>>>>>d
>>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.
>>>>>>0.
>>>>>>0%
>>>>>>t
>>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>
>>>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>>>eth1mask.
>>>>>>
>>>>>>Thanks
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> I don’t think its relevant as the piece we’ve fixed, just
>>>>>>>eliminated
>>>>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>>>>eth1
>>>>>>> doesn’t seem related to the problem (although we have to figure out
>>>>>>>why
>>>>>>> its missing), as the connection that fails, happening to link local
>>>>>>>169.x
>>>>>>> eth0 interface.
>>>>>>>
>>>>>>> Edison, can you please tell us how to debug link local connection
>>>>>>>failure,
>>>>>>> on KVM agent?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Alena.
>>>>>>>
>>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>thank you, the NullPointerException doesn't occur any more, but
>>>>>>>>there
>>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>>
>>>>>>>>When I start the virtual router, it comes up, but in agent.log,
>>>>>>>>there
>>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>>
>>>>>>>>Then I logged into the virtual router to find out that the netmask
>>>>>>>>of
>>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>>
>>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>>Connected to domain r-19-VM
>>>>>>>>Escape character is ^]
>>>>>>>>
>>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>>
>>>>>>>>r-19-VM login: root
>>>>>>>>...
>>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>>iface lo inet loopback
>>>>>>>>
>>>>>>>>iface eth0 inet static
>>>>>>>> address 169.254.1.59
>>>>>>>> netmask 255.255.0.0
>>>>>>>>iface eth1 inet static
>>>>>>>> address 10.193.17.1
>>>>>>>> netmask
>>>>>>>>iface eth2 inet static
>>>>>>>> address 10.193.17.190
>>>>>>>> netmask 255.255.255.0
>>>>>>>>
>>>>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>>>>where the parameters are visible:
>>>>>>>>
>>>>>>>>2014-03-24 21:36:17,681 DEBUG
>>>>>>>>[kvm.resource.LibvirtComputingResource]
>>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.s
>>>>>>>>h
>>>>>>>>-l
>>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.2
>>>>>>>>55
>>>>>>>>.0
>>>>>>>>%
>>>>>>>>ga
>>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=
>>>>>>>>cs
>>>>>>>>6c
>>>>>>>>l
>>>>>>>>ou
>>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.2
>>>>>>>>55
>>>>>>>>.0
>>>>>>>>.
>>>>>>>>0%
>>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>>
>>>>>>>>
>>>>>>>>Any hint is appreciated.
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit.
>>>>>>>>>If
>>>>>>>>>vm
>>>>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>>
>>>>>>>>> 1) Save the db dump.
>>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>>
>>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null
>>>>>>>>>where
>>>>>>>>> id=<problematic public ip address id>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Let me know how it works.
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>>>>query
>>>>>>>>>>correctly:
>>>>>>>>>>
>>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>>from user_ip_address uip
>>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr
>>>>>>>>>>where
>>>>>>>>>>fr.network_id=205);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>>nic_removed
>>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>>2014-03-18
>>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>>>> | Stopped | NULL |
>>>>>>>>>>+-------+------------+-------------------+--------------+--------
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+-----------+------------+
>>>>>>>>>>
>>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>>
>>>>>>>>>>Thank you
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>>
>>>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>>>right?)
>>>>>>>>>>>
>>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>>network_id=205;
>>>>>>>>>>>
>>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>>>>
>>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in
>>>>>>>>>>>(Select
>>>>>>>>>>>id,
>>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>>
>>>>>>>>>>> For each vmId/networkId combo, check if there is non-removed
>>>>>>>>>>>nic
>>>>>>>>>>>and
>>>>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>>>>reference
>>>>>>>>>>> referring to vm that is removed already. If you find any, let
>>>>>>>>>>>me
>>>>>>>>>>>know
>>>>>>>>>>>and
>>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>>
>>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>>
>>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but
>>>>>>>>>>>>I
>>>>>>>>>>>>removed
>>>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>>>created
>>>>>>>>>>>>which also couldn't be started due to the same NPE. I reverted
>>>>>>>>>>>>the
>>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>>
>>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>>router
>>>>>>>>>>>>which
>>>>>>>>>>>>was used by my instances. I deleted the router using the UI
>>>>>>>>>>>>after
>>>>>>>>>>>>the
>>>>>>>>>>>>first
>>>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>>>suggested
>>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>>procedure
>>>>>>>>>>>>solved the problem).
>>>>>>>>>>>>
>>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>>
>>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>>
>>>>>>>>>>>>Thank you
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>
>>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>>n.gateway,
>>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>>>>i.name,
>>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on
>>>>>>>>>>>>n.instance_id =
>>>>>>>>>>>>i.id
>>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>>>gateway
>>>>>>>>>>>>| ip_type | reserver_name | network_id |
>>>>>>>>>>>>instance_id |
>>>>>>>>>>>>name
>>>>>>>>>>>>| state | type |
>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 |
>>>>>>>>>>>>4 |
>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>4 |
>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>4 |
>>>>>>>>>>>>r-4-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>7 |
>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>7 |
>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>7 |
>>>>>>>>>>>>r-7-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>18 |
>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>18 |
>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>18 |
>>>>>>>>>>>>r-18-VM
>>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 |
>>>>>>>>>>>>19 |
>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>| 30 | NULL | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 |
>>>>>>>>>>>>19 |
>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 |
>>>>>>>>>>>>19 |
>>>>>>>>>>>>r-19-VM
>>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>>+----+---------------------+---------------+---------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>---------+--------------------------+------------+-------------
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>>
>>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The error happens not because Ip is null, but because the nic
>>>>>>>>>>>>>in
>>>>>>>>>>>>>a
>>>>>>>>>>>>>certain
>>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC
>>>>>>>>>>>>>nic
>>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>>>back.
>>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>>
>>>>>>>>>>>>> select * from router_network_ref where router_id=<id of your
>>>>>>>>>>>>>VR)
>>>>>>>>>>>>>and
>>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>>of
>>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>>
>>>>>>>>>>>>> It will give you the list of networks refs that somehow
>>>>>>>>>>>>>weren¹t
>>>>>>>>>>>>>cleaned
>>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>>> table.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>as this is my first post to the list, I would like to thank
>>>>>>>>>>>>>>all
>>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall
>>>>>>>>>>>>>>without
>>>>>>>>>>>>>>any
>>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>After a restart of the management server (stopping and
>>>>>>>>>>>>>>starting
>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>java
>>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>>>found
>>>>>>>>>>>>>>out
>>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL in
>>>>>>>>>>>>>>ip
>>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>>row
>>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>with
>>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not
>>>>>>>>>>>>>>supported
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>the
>>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202
>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already
>>>>>>>>>>>>>>implemented
>>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202
>>>>>>>>>>>>>>as a
>>>>>>>>>>>>>>part
>>>>>>>>>>>>>>of
>>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare
>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>>>associated
>>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>>handle
>>>>>>>>>>>>>>service
>>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>>for
>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned:
>>>>>>>>>>>>>>200,
>>>>>>>>>>>>>>adding
>>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of
>>>>>>>>>>>>>>domR
>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding,
>>>>>>>>>>>>>>load
>>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>>>>apply
>>>>>>>>>>>>>>as
>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkMode
>>>>>>>>>>>>>>lI
>>>>>>>>>>>>>>mp
>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>76
>>>>>>>>>>>>>>3)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>>tw
>>>>>>>>>>>>>>or
>>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346
>>>>>>>>>>>>>>)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImp
>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>eN
>>>>>>>>>>>>>>et
>>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.jav
>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>92
>>>>>>>>>>>>>>8
>>>>>>>>>>>>>>)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>Co
>>>>>>>>>>>>>>mm
>>>>>>>>>>>>>>an
>>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImp
>>>>>>>>>>>>>>l.
>>>>>>>>>>>>>>fi
>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>iz
>>>>>>>>>>>>>>eC
>>>>>>>>>>>>>>om
>>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.f
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>ze
>>>>>>>>>>>>>>De
>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>oy
>>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMa
>>>>>>>>>>>>>>ch
>>>>>>>>>>>>>>in
>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMa
>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>r
>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>(
>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>lN
>>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>V
>>>>>>>>>>>>>>ir
>>>>>>>>>>>>>>tu
>>>>>>>>>>>>>>al
>>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>rs
>>>>>>>>>>>>>>(V
>>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.d
>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>lo
>>>>>>>>>>>>>>y
>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1
>>>>>>>>>>>>>>90
>>>>>>>>>>>>>>2)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Virt
>>>>>>>>>>>>>>ua
>>>>>>>>>>>>>>lR
>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>El
>>>>>>>>>>>>>>em
>>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElements
>>>>>>>>>>>>>>An
>>>>>>>>>>>>>>dR
>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>so
>>>>>>>>>>>>>>ur
>>>>>>>>>>>>>>ce
>>>>>>>>>>>>>>s(
>>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(Network
>>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>>na
>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>Im
>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor
>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:
>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkMana
>>>>>>>>>>>>>>ge
>>>>>>>>>>>>>>rI
>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>pl
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>>av
>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>2435)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.s
>>>>>>>>>>>>>>ta
>>>>>>>>>>>>>>rt
>>>>>>>>>>>>>>R
>>>>>>>>>>>>>>ou
>>>>>>>>>>>>>>te
>>>>>>>>>>>>>>r(
>>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor
>>>>>>>>>>>>>>$I
>>>>>>>>>>>>>>nt
>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>rc
>>>>>>>>>>>>>>ep
>>>>>>>>>>>>>>to
>>>>>>>>>>>>>>rD
>>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:
>>>>>>>>>>>>>>12
>>>>>>>>>>>>>>5)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>>>>>>>>>>>>.e
>>>>>>>>>>>>>>xe
>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>ut
>>>>>>>>>>>>>>e(
>>>>>>>>>>>>>>St
>>>>>>>>>>>>>>ar
>>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>>| network_id | mode | state | strategy |
>>>>>>>>>>>>>>reserver_name
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>reservation_id | device_id |
>>>>>>>>>>>>>>update_time
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>created
>>>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 |
>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 |
>>>>>>>>>>>>>>169.254.0.1 |
>>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>>0
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>|
>>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL |
>>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>+----+--------------------------------------+-------------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+------------+--------+--------------+----------+------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-+
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------------------------------+-----------+--------------
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>----------+-------------+-------------+--------------------+-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>--
>>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
select id, name, traffic_type, broadcast_domain_type, cidr, gateway, mode, state, removed from networks where id = 205;
+-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+
| id | name | traffic_type | broadcast_domain_type | cidr | gateway | mode | state | removed |
+-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+
| 205 | default | Guest | Vlan | 10.124.99.0/24 | 10.124.99.1 | Dhcp | Allocated | NULL |
+-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+
Cheers
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> No, it doesn’t seem right to me having 2 nics with device id 0. But looks
> like they’ve got programmed to correct devices on the backend per your
> prev email?
>
> iface eth0 inet static
> address 169.254.1.59
> netmask 255.255.0.0
>
> iface eth1 inet static
> address 10.193.17.1
> Netmask
>
> iface eth2 inet static
> address 10.193.17.190
> netmask 255.255.255.0
>
>
>
> I can see that only one parameter is missing from the start command, the
> second nic (network id=205) doesn’t have the gateway.
> From the command/DB, I see that the gateway is missing in the nics table
> for the network 205? Can you check gateway information in the networks
> table for the id=205
>
>
>
>
> On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>select
>>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>>from nics where instance_id=19;
>>
>>+----+---------------+---------------+-------------+-----------+---------+
>>------------+--------------------------+
>>| id | ip4_address | netmask | gateway | state | removed
>>| network_id | reserver_name |
>>+----+---------------+---------------+-------------+-----------+---------+
>>------------+--------------------------+
>>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>>| 205 | ExternalGuestNetworkGuru |
>>| 30 | NULL | NULL | NULL | Allocated | NULL
>>| 202 | ControlNetworkGuru |
>>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>>| 200 | PublicNetworkGuru |
>>+----+---------------+---------------+-------------+-----------+---------+
>>------------+--------------------------+
>>
>>and this is the nics element from the StartCmd. Is it normal to have
>>two nics with deviceId 0?
>>
>>"nics":[
>> {"deviceId":2,
>> "networkRateMbps":200,
>> "defaultNic":true,
>> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
>> "ip":"10.193.17.190",
>> "netmask":"255.255.255.0",
>> "gateway":"10.193.17.1",
>> "mac":"06:7e:fe:00:00:bf",
>> "dns1":"10.193.17.1",
>> "broadcastType":"Vlan",
>> "type":"Public",
>> "broadcastUri":"vlan://untagged",
>> "isolationUri":"vlan://untagged",
>> "isSecurityGroupEnabled":false,
>> "name":"cloudbr0"},
>> {"deviceId":0,
>> "networkRateMbps":200,
>> "defaultNic":false,
>> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
>> "ip":"10.124.99.1",
>> "netmask":"255.255.255.0",
>> "mac":"02:00:2a:43:00:0d",
>> "dns1":"10.193.17.1",
>> "broadcastType":"Vlan",
>> "type":"Guest",
>> "broadcastUri":"vlan://3949",
>> "isolationUri":"vlan://3949",
>> "isSecurityGroupEnabled":false,
>> "name":"cloudbr1"},
>> {"deviceId":0,
>> "networkRateMbps":-1,
>> "defaultNic":false,
>> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
>> "ip":"169.254.1.59",
>> "netmask":"255.255.0.0",
>> "gateway":"169.254.0.1",
>> "mac":"0e:00:a9:fe:01:3b",
>> "broadcastType":"LinkLocal",
>> "type":"Control",
>> "isSecurityGroupEnabled":false}
>>]
>>
>>Thanks
>>
>>
>>Kambiz
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> Kambiz, the debug statements below are for the case when eth1 is a
>>>control
>>> interface as it was in your old command. I’ve looked at the new command,
>>> eth1 is not control, its either public or guest
>>>
>>> eth0: - control
>>>
>>> iface eth0 inet static
>>> address 169.254.1.59
>>> netmask 255.255.0.0
>>>
>>> eth1:
>>>
>>> iface eth1 inet static
>>> address 10.193.17.1
>>> Netmask
>>>
>>> So you need to execute the mysql statements for the traffic type of VR
>>>nic
>>> eth1
>>>
>>> -Alena.
>>>
>>>
>>>
>>>
>>> On 3/25/14, 9:57 AM, "Alena Prokharchyk" <Al...@citrix.com>
>>> wrote:
>>>
>>>>Kambiz, can you please check the following:
>>>>
>>>>
>>>>1) Check if the gateway is set on control network:
>>>>
>>>>mysql> select gateway, cidr from networks where traffic_type=‘Control’;
>>>>
>>>>2) For router control nic, check if network/gateway are set.
>>>>
>>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>>network_id=<control network id>
>>>>
>>>>-Alena.
>>>>
>>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>I looked up the startup command of the old router instance which worked
>>>>>correctly:
>>>>>
>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%
>>>>>ga
>>>>>t
>>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl
>>>>>ou
>>>>>d
>>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.
>>>>>0%
>>>>>t
>>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>
>>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>>eth1mask.
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>>>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>>>eth1
>>>>>> doesn’t seem related to the problem (although we have to figure out
>>>>>>why
>>>>>> its missing), as the connection that fails, happening to link local
>>>>>>169.x
>>>>>> eth0 interface.
>>>>>>
>>>>>> Edison, can you please tell us how to debug link local connection
>>>>>>failure,
>>>>>> on KVM agent?
>>>>>>
>>>>>> Thank you,
>>>>>> Alena.
>>>>>>
>>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>thank you, the NullPointerException doesn't occur any more, but there
>>>>>>>still seems to be a problem during startup of the router.
>>>>>>>
>>>>>>>When I start the virtual router, it comes up, but in agent.log, there
>>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>>
>>>>>>>Then I logged into the virtual router to find out that the netmask of
>>>>>>>eth1 is missing in the interfaces file:
>>>>>>>
>>>>>>>root@host:~# virsh console r-19-VM
>>>>>>>Connected to domain r-19-VM
>>>>>>>Escape character is ^]
>>>>>>>
>>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>>
>>>>>>>r-19-VM login: root
>>>>>>>...
>>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>>auto lo eth0 eth1 eth2
>>>>>>>iface lo inet loopback
>>>>>>>
>>>>>>>iface eth0 inet static
>>>>>>> address 169.254.1.59
>>>>>>> netmask 255.255.0.0
>>>>>>>iface eth1 inet static
>>>>>>> address 10.193.17.1
>>>>>>> netmask
>>>>>>>iface eth2 inet static
>>>>>>> address 10.193.17.190
>>>>>>> netmask 255.255.255.0
>>>>>>>
>>>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>>>where the parameters are visible:
>>>>>>>
>>>>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh
>>>>>>>-l
>>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255
>>>>>>>.0
>>>>>>>%
>>>>>>>ga
>>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs
>>>>>>>6c
>>>>>>>l
>>>>>>>ou
>>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255
>>>>>>>.0
>>>>>>>.
>>>>>>>0%
>>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>>
>>>>>>>
>>>>>>>Any hint is appreciated.
>>>>>>>
>>>>>>>Thanks
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If
>>>>>>>>vm
>>>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>>
>>>>>>>> 1) Save the db dump.
>>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>>
>>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>>>>>> id=<problematic public ip address id>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know how it works.
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>>>query
>>>>>>>>>correctly:
>>>>>>>>>
>>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>>from user_ip_address uip
>>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>>>>>fr.network_id=205);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>>--
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+-----------+------------+
>>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>>nic_removed
>>>>>>>>> | vm_state | vm_removed |
>>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>>--
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+-----------+------------+
>>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>>2014-03-18
>>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>>> | Stopped | NULL |
>>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>>--
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+-----------+------------+
>>>>>>>>>
>>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>>
>>>>>>>>>Thank you
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>>
>>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>>right?)
>>>>>>>>>>
>>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>>network_id=205;
>>>>>>>>>>
>>>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>>>
>>>>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>>>>>id,
>>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>>
>>>>>>>>>> For each vmId/networkId combo, check if there is non-removed nic
>>>>>>>>>>and
>>>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>>>reference
>>>>>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>>>>>know
>>>>>>>>>>and
>>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi Alena,
>>>>>>>>>>>
>>>>>>>>>>>thank you for your help.
>>>>>>>>>>>
>>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>>>>>removed
>>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>>created
>>>>>>>>>>>which also couldn't be started due to the same NPE. I reverted
>>>>>>>>>>>the
>>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>>
>>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>>router
>>>>>>>>>>>which
>>>>>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>>>>>the
>>>>>>>>>>>first
>>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>>suggested
>>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>>procedure
>>>>>>>>>>>solved the problem).
>>>>>>>>>>>
>>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>>
>>>>>>>>>>>Anything else I can try?
>>>>>>>>>>>
>>>>>>>>>>>Thank you
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>>n.gateway,
>>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>>>i.name,
>>>>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>>>>>i.id
>>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>>gateway
>>>>>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>>>>>name
>>>>>>>>>>>| state | type |
>>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>>>>>r-4-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>>>>>r-4-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>>>>>r-4-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>>>>>r-7-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>>>>>r-7-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>>>>>r-7-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>>>>>r-18-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>>>>>r-18-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>>>>>r-18-VM
>>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>>>>>r-19-VM
>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>>>>>r-19-VM
>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>>10.193.17.1
>>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>>>>>r-19-VM
>>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>>--
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>-----------+--------------+
>>>>>>>>>>>
>>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> The error happens not because Ip is null, but because the nic
>>>>>>>>>>>>in
>>>>>>>>>>>>a
>>>>>>>>>>>>certain
>>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>>
>>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>>back.
>>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>>
>>>>>>>>>>>> select * from router_network_ref where router_id=<id of your
>>>>>>>>>>>>VR)
>>>>>>>>>>>>and
>>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>>of
>>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>>
>>>>>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>>>>>cleaned
>>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>>router_network_ref
>>>>>>>>>>>> table.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>>
>>>>>>>>>>>> -Alena.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>>>>>any
>>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>>
>>>>>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>>>>>the
>>>>>>>>>>>>>java
>>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>>
>>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>>found
>>>>>>>>>>>>>out
>>>>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>>>>>(cf.
>>>>>>>>>>>>>row
>>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>>
>>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>>>suggestions
>>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>>
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>for
>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>>associated
>>>>>>>>>>>>>with
>>>>>>>>>>>>>any VPC
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>handle
>>>>>>>>>>>>>service
>>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported
>>>>>>>>>>>>>in
>>>>>>>>>>>>>the
>>>>>>>>>>>>>network id=205
>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202
>>>>>>>>>>>>>as a
>>>>>>>>>>>>>part
>>>>>>>>>>>>>of
>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202
>>>>>>>>>>>>>as a
>>>>>>>>>>>>>part
>>>>>>>>>>>>>of
>>>>>>>>>>>>>network implement
>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>>prepare
>>>>>>>>>>>>>for
>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>>associated
>>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>>handle
>>>>>>>>>>>>>service
>>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>>volumes
>>>>>>>>>>>>>for
>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>>>>>adding
>>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of
>>>>>>>>>>>>>domR
>>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding,
>>>>>>>>>>>>>load
>>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>>>apply
>>>>>>>>>>>>>as
>>>>>>>>>>>>>a
>>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelI
>>>>>>>>>>>>>mp
>>>>>>>>>>>>>l
>>>>>>>>>>>>>.j
>>>>>>>>>>>>>av
>>>>>>>>>>>>>a:
>>>>>>>>>>>>>76
>>>>>>>>>>>>>3)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>>al
>>>>>>>>>>>>>i
>>>>>>>>>>>>>ze
>>>>>>>>>>>>>Ne
>>>>>>>>>>>>>tw
>>>>>>>>>>>>>or
>>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.
>>>>>>>>>>>>>fi
>>>>>>>>>>>>>n
>>>>>>>>>>>>>al
>>>>>>>>>>>>>iz
>>>>>>>>>>>>>eN
>>>>>>>>>>>>>et
>>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>>92
>>>>>>>>>>>>>8
>>>>>>>>>>>>>)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>>al
>>>>>>>>>>>>>i
>>>>>>>>>>>>>ze
>>>>>>>>>>>>>Co
>>>>>>>>>>>>>mm
>>>>>>>>>>>>>an
>>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.
>>>>>>>>>>>>>fi
>>>>>>>>>>>>>n
>>>>>>>>>>>>>al
>>>>>>>>>>>>>iz
>>>>>>>>>>>>>eC
>>>>>>>>>>>>>om
>>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>>al
>>>>>>>>>>>>>i
>>>>>>>>>>>>>ze
>>>>>>>>>>>>>De
>>>>>>>>>>>>>pl
>>>>>>>>>>>>>oy
>>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMach
>>>>>>>>>>>>>in
>>>>>>>>>>>>>e
>>>>>>>>>>>>>Ma
>>>>>>>>>>>>>na
>>>>>>>>>>>>>ge
>>>>>>>>>>>>>rI
>>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMana
>>>>>>>>>>>>>ge
>>>>>>>>>>>>>r
>>>>>>>>>>>>>Im
>>>>>>>>>>>>>pl
>>>>>>>>>>>>>.j
>>>>>>>>>>>>>av
>>>>>>>>>>>>>a:471)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>(
>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>ua
>>>>>>>>>>>>>lN
>>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>V
>>>>>>>>>>>>>ir
>>>>>>>>>>>>>tu
>>>>>>>>>>>>>al
>>>>>>>>>>>>>Ro
>>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>R
>>>>>>>>>>>>>ou
>>>>>>>>>>>>>te
>>>>>>>>>>>>>rs
>>>>>>>>>>>>>(V
>>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.dep
>>>>>>>>>>>>>lo
>>>>>>>>>>>>>y
>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>ua
>>>>>>>>>>>>>lR
>>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:190
>>>>>>>>>>>>>2)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Virtua
>>>>>>>>>>>>>lR
>>>>>>>>>>>>>o
>>>>>>>>>>>>>ut
>>>>>>>>>>>>>er
>>>>>>>>>>>>>El
>>>>>>>>>>>>>em
>>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAn
>>>>>>>>>>>>>dR
>>>>>>>>>>>>>e
>>>>>>>>>>>>>so
>>>>>>>>>>>>>ur
>>>>>>>>>>>>>ce
>>>>>>>>>>>>>s(
>>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMa
>>>>>>>>>>>>>na
>>>>>>>>>>>>>g
>>>>>>>>>>>>>er
>>>>>>>>>>>>>Im
>>>>>>>>>>>>>pl
>>>>>>>>>>>>>.j
>>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I
>>>>>>>>>>>>>nt
>>>>>>>>>>>>>e
>>>>>>>>>>>>>rc
>>>>>>>>>>>>>ep
>>>>>>>>>>>>>to
>>>>>>>>>>>>>rD
>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12
>>>>>>>>>>>>>5)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManage
>>>>>>>>>>>>>rI
>>>>>>>>>>>>>m
>>>>>>>>>>>>>pl
>>>>>>>>>>>>>.j
>>>>>>>>>>>>>av
>>>>>>>>>>>>>a:
>>>>>>>>>>>>>2435)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>R
>>>>>>>>>>>>>ou
>>>>>>>>>>>>>te
>>>>>>>>>>>>>r(
>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>>rt
>>>>>>>>>>>>>R
>>>>>>>>>>>>>ou
>>>>>>>>>>>>>te
>>>>>>>>>>>>>r(
>>>>>>>>>>>>>Vi
>>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I
>>>>>>>>>>>>>nt
>>>>>>>>>>>>>e
>>>>>>>>>>>>>rc
>>>>>>>>>>>>>ep
>>>>>>>>>>>>>to
>>>>>>>>>>>>>rD
>>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12
>>>>>>>>>>>>>5)
>>>>>>>>>>>>> at
>>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.e
>>>>>>>>>>>>>xe
>>>>>>>>>>>>>c
>>>>>>>>>>>>>ut
>>>>>>>>>>>>>e(
>>>>>>>>>>>>>St
>>>>>>>>>>>>>ar
>>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>table nics:
>>>>>>>>>>>>>
>>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>>mac_address
>>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>>>>>> |
>>>>>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>>>>>> |
>>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>>|
>>>>>>>>>>>>>created
>>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>|
>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>>0
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>|
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>>>Ip4
>>>>>>>>>>>>>|
>>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>>0
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>>0 |
>>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>>|
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>>> |
>>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>>> |
>>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>>> |
>>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>>NULL
>>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>>|
>>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>>>>>> |
>>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-+
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>+-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>--
>>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
No, it doesn’t seem right to me having 2 nics with device id 0. But looks
like they’ve got programmed to correct devices on the backend per your
prev email?
iface eth0 inet static
address 169.254.1.59
netmask 255.255.0.0
iface eth1 inet static
address 10.193.17.1
Netmask
iface eth2 inet static
address 10.193.17.190
netmask 255.255.255.0
I can see that only one parameter is missing from the start command, the
second nic (network id=205) doesn’t have the gateway.
From the command/DB, I see that the gateway is missing in the nics table
for the network 205? Can you check gateway information in the networks
table for the id=205
On 3/25/14, 1:01 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>select
>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
>from nics where instance_id=19;
>
>+----+---------------+---------------+-------------+-----------+---------+
>------------+--------------------------+
>| id | ip4_address | netmask | gateway | state | removed
>| network_id | reserver_name |
>+----+---------------+---------------+-------------+-----------+---------+
>------------+--------------------------+
>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL
>| 205 | ExternalGuestNetworkGuru |
>| 30 | NULL | NULL | NULL | Allocated | NULL
>| 202 | ControlNetworkGuru |
>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL
>| 200 | PublicNetworkGuru |
>+----+---------------+---------------+-------------+-----------+---------+
>------------+--------------------------+
>
>and this is the nics element from the StartCmd. Is it normal to have
>two nics with deviceId 0?
>
>"nics":[
> {"deviceId":2,
> "networkRateMbps":200,
> "defaultNic":true,
> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
> "ip":"10.193.17.190",
> "netmask":"255.255.255.0",
> "gateway":"10.193.17.1",
> "mac":"06:7e:fe:00:00:bf",
> "dns1":"10.193.17.1",
> "broadcastType":"Vlan",
> "type":"Public",
> "broadcastUri":"vlan://untagged",
> "isolationUri":"vlan://untagged",
> "isSecurityGroupEnabled":false,
> "name":"cloudbr0"},
> {"deviceId":0,
> "networkRateMbps":200,
> "defaultNic":false,
> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
> "ip":"10.124.99.1",
> "netmask":"255.255.255.0",
> "mac":"02:00:2a:43:00:0d",
> "dns1":"10.193.17.1",
> "broadcastType":"Vlan",
> "type":"Guest",
> "broadcastUri":"vlan://3949",
> "isolationUri":"vlan://3949",
> "isSecurityGroupEnabled":false,
> "name":"cloudbr1"},
> {"deviceId":0,
> "networkRateMbps":-1,
> "defaultNic":false,
> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
> "ip":"169.254.1.59",
> "netmask":"255.255.0.0",
> "gateway":"169.254.0.1",
> "mac":"0e:00:a9:fe:01:3b",
> "broadcastType":"LinkLocal",
> "type":"Control",
> "isSecurityGroupEnabled":false}
>]
>
>Thanks
>
>
>Kambiz
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> Kambiz, the debug statements below are for the case when eth1 is a
>>control
>> interface as it was in your old command. I’ve looked at the new command,
>> eth1 is not control, its either public or guest
>>
>> eth0: - control
>>
>> iface eth0 inet static
>> address 169.254.1.59
>> netmask 255.255.0.0
>>
>> eth1:
>>
>> iface eth1 inet static
>> address 10.193.17.1
>> Netmask
>>
>> So you need to execute the mysql statements for the traffic type of VR
>>nic
>> eth1
>>
>> -Alena.
>>
>>
>>
>>
>> On 3/25/14, 9:57 AM, "Alena Prokharchyk" <Al...@citrix.com>
>> wrote:
>>
>>>Kambiz, can you please check the following:
>>>
>>>
>>>1) Check if the gateway is set on control network:
>>>
>>>mysql> select gateway, cidr from networks where traffic_type=‘Control’;
>>>
>>>2) For router control nic, check if network/gateway are set.
>>>
>>>Select gateway,netmask from nics where instance_id=<routerid> and
>>>network_id=<control network id>
>>>
>>>-Alena.
>>>
>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>I looked up the startup command of the old router instance which worked
>>>>correctly:
>>>>
>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%
>>>>ga
>>>>t
>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl
>>>>ou
>>>>d
>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.
>>>>0%
>>>>t
>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>
>>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>>eth1mask.
>>>>
>>>>Thanks
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>>eth1
>>>>> doesn’t seem related to the problem (although we have to figure out
>>>>>why
>>>>> its missing), as the connection that fails, happening to link local
>>>>>169.x
>>>>> eth0 interface.
>>>>>
>>>>> Edison, can you please tell us how to debug link local connection
>>>>>failure,
>>>>> on KVM agent?
>>>>>
>>>>> Thank you,
>>>>> Alena.
>>>>>
>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>thank you, the NullPointerException doesn't occur any more, but there
>>>>>>still seems to be a problem during startup of the router.
>>>>>>
>>>>>>When I start the virtual router, it comes up, but in agent.log, there
>>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>>
>>>>>>Then I logged into the virtual router to find out that the netmask of
>>>>>>eth1 is missing in the interfaces file:
>>>>>>
>>>>>>root@host:~# virsh console r-19-VM
>>>>>>Connected to domain r-19-VM
>>>>>>Escape character is ^]
>>>>>>
>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>>
>>>>>>r-19-VM login: root
>>>>>>...
>>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>>auto lo eth0 eth1 eth2
>>>>>>iface lo inet loopback
>>>>>>
>>>>>>iface eth0 inet static
>>>>>> address 169.254.1.59
>>>>>> netmask 255.255.0.0
>>>>>>iface eth1 inet static
>>>>>> address 10.193.17.1
>>>>>> netmask
>>>>>>iface eth2 inet static
>>>>>> address 10.193.17.190
>>>>>> netmask 255.255.255.0
>>>>>>
>>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>>where the parameters are visible:
>>>>>>
>>>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh
>>>>>>-l
>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255
>>>>>>.0
>>>>>>%
>>>>>>ga
>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs
>>>>>>6c
>>>>>>l
>>>>>>ou
>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255
>>>>>>.0
>>>>>>.
>>>>>>0%
>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>>
>>>>>>
>>>>>>Any hint is appreciated.
>>>>>>
>>>>>>Thanks
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If
>>>>>>>vm
>>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>>> user_ip_address table. Here is the flow:
>>>>>>>
>>>>>>> 1) Save the db dump.
>>>>>>> 2) Run the query to cleanup the reference:
>>>>>>>
>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>>>>> id=<problematic public ip address id>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Let me know how it works.
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>>query
>>>>>>>>correctly:
>>>>>>>>
>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>>from user_ip_address uip
>>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>>>>fr.network_id=205);
>>>>>>>>
>>>>>>>>
>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>--
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+-----------+------------+
>>>>>>>>| vm_id | network_id | public_ip_address | nic_state |
>>>>>>>>nic_removed
>>>>>>>> | vm_state | vm_removed |
>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>--
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+-----------+------------+
>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating |
>>>>>>>>2014-03-18
>>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>>> | Stopped | NULL |
>>>>>>>>+-------+------------+-------------------+--------------+----------
>>>>>>>>--
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+-----------+------------+
>>>>>>>>
>>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>>
>>>>>>>>Thank you
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>>
>>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>>right?)
>>>>>>>>>
>>>>>>>>> Select id, ip_address_id from firewall_rules where
>>>>>>>>>network_id=205;
>>>>>>>>>
>>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>>
>>>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>>>>id,
>>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>>
>>>>>>>>> For each vmId/networkId combo, check if there is non-removed nic
>>>>>>>>>and
>>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>>reference
>>>>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>>>>know
>>>>>>>>>and
>>>>>>>>> I will tell you how to clean it up
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi Alena,
>>>>>>>>>>
>>>>>>>>>>thank you for your help.
>>>>>>>>>>
>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>>>>removed
>>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>>created
>>>>>>>>>>which also couldn't be started due to the same NPE. I reverted
>>>>>>>>>>the
>>>>>>>>>>change by restoring from the dump.
>>>>>>>>>>
>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the
>>>>>>>>>>router
>>>>>>>>>>which
>>>>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>>>>the
>>>>>>>>>>first
>>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>>suggested
>>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>>procedure
>>>>>>>>>>solved the problem).
>>>>>>>>>>
>>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>>
>>>>>>>>>>Anything else I can try?
>>>>>>>>>>
>>>>>>>>>>Thank you
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask,
>>>>>>>>>>n.gateway,
>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>>i.name,
>>>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>>>>i.id
>>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>-----------+--------------+
>>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>>gateway
>>>>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>>>>name
>>>>>>>>>>| state | type |
>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>-----------+--------------+
>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>>>>r-4-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>>>>r-4-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>>10.193.17.1
>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>>>>r-4-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>>>>r-7-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>>>>r-7-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>10.193.17.1
>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>>>>r-7-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>>>>r-18-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>>>>r-18-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>10.193.17.1
>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>>>>r-18-VM
>>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>>>>r-19-VM
>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>>>>r-19-VM
>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>>10.193.17.1
>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>>>>r-19-VM
>>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>>+----+---------------------+---------------+---------------+-----
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>---------+--------------------------+------------+-------------+-
>>>>>>>>>>--
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>-----------+--------------+
>>>>>>>>>>
>>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> The error happens not because Ip is null, but because the nic
>>>>>>>>>>>in
>>>>>>>>>>>a
>>>>>>>>>>>certain
>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>>
>>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>>
>>>>>>>>>>> 1) Stop the MS
>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert
>>>>>>>>>>>back.
>>>>>>>>>>> 3) Run the query:
>>>>>>>>>>>
>>>>>>>>>>> select * from router_network_ref where router_id=<id of your
>>>>>>>>>>>VR)
>>>>>>>>>>>and
>>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>>instance_id=<ID
>>>>>>>>>>>of
>>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>>
>>>>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>>>>cleaned
>>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>>router_network_ref
>>>>>>>>>>> table.
>>>>>>>>>>>
>>>>>>>>>>> Let me know how it works.
>>>>>>>>>>>
>>>>>>>>>>> -Alena.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>>wrote:
>>>>>>>>>>>
>>>>>>>>>>>>Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>>>>any
>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>>
>>>>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>>>>the
>>>>>>>>>>>>java
>>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>>
>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I
>>>>>>>>>>>>found
>>>>>>>>>>>>out
>>>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>>>>(cf.
>>>>>>>>>>>>row
>>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>>
>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>>suggestions
>>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>>
>>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Kambiz
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>management-server.log:
>>>>>>>>>>>>
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>prepare
>>>>>>>>>>>>for
>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>>associated
>>>>>>>>>>>>with
>>>>>>>>>>>>any VPC
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG
>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>handle
>>>>>>>>>>>>service
>>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported
>>>>>>>>>>>>in
>>>>>>>>>>>>the
>>>>>>>>>>>>network id=205
>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202
>>>>>>>>>>>>as a
>>>>>>>>>>>>part
>>>>>>>>>>>>of
>>>>>>>>>>>>network implement
>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202
>>>>>>>>>>>>as a
>>>>>>>>>>>>part
>>>>>>>>>>>>of
>>>>>>>>>>>>network implement
>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to
>>>>>>>>>>>>prepare
>>>>>>>>>>>>for
>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>>associated
>>>>>>>>>>>>with any VPC
>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>[cloud.network.NetworkManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG
>>>>>>>>>>>>[network.element.NiciraNvpElement]
>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can
>>>>>>>>>>>>handle
>>>>>>>>>>>>service
>>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1
>>>>>>>>>>>>volumes
>>>>>>>>>>>>for
>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG
>>>>>>>>>>>>[cloud.storage.StorageManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>>>>adding
>>>>>>>>>>>>disk to VM
>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for
>>>>>>>>>>>>VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of
>>>>>>>>>>>>domR
>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding,
>>>>>>>>>>>>load
>>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>>apply
>>>>>>>>>>>>as
>>>>>>>>>>>>a
>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelI
>>>>>>>>>>>>mp
>>>>>>>>>>>>l
>>>>>>>>>>>>.j
>>>>>>>>>>>>av
>>>>>>>>>>>>a:
>>>>>>>>>>>>76
>>>>>>>>>>>>3)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>al
>>>>>>>>>>>>i
>>>>>>>>>>>>ze
>>>>>>>>>>>>Ne
>>>>>>>>>>>>tw
>>>>>>>>>>>>or
>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.
>>>>>>>>>>>>fi
>>>>>>>>>>>>n
>>>>>>>>>>>>al
>>>>>>>>>>>>iz
>>>>>>>>>>>>eN
>>>>>>>>>>>>et
>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:
>>>>>>>>>>>>92
>>>>>>>>>>>>8
>>>>>>>>>>>>)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>al
>>>>>>>>>>>>i
>>>>>>>>>>>>ze
>>>>>>>>>>>>Co
>>>>>>>>>>>>mm
>>>>>>>>>>>>an
>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.
>>>>>>>>>>>>fi
>>>>>>>>>>>>n
>>>>>>>>>>>>al
>>>>>>>>>>>>iz
>>>>>>>>>>>>eC
>>>>>>>>>>>>om
>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>>>>al
>>>>>>>>>>>>i
>>>>>>>>>>>>ze
>>>>>>>>>>>>De
>>>>>>>>>>>>pl
>>>>>>>>>>>>oy
>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMach
>>>>>>>>>>>>in
>>>>>>>>>>>>e
>>>>>>>>>>>>Ma
>>>>>>>>>>>>na
>>>>>>>>>>>>ge
>>>>>>>>>>>>rI
>>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMana
>>>>>>>>>>>>ge
>>>>>>>>>>>>r
>>>>>>>>>>>>Im
>>>>>>>>>>>>pl
>>>>>>>>>>>>.j
>>>>>>>>>>>>av
>>>>>>>>>>>>a:471)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>rt
>>>>>>>>>>>>(
>>>>>>>>>>>>Vi
>>>>>>>>>>>>rt
>>>>>>>>>>>>ua
>>>>>>>>>>>>lN
>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>rt
>>>>>>>>>>>>V
>>>>>>>>>>>>ir
>>>>>>>>>>>>tu
>>>>>>>>>>>>al
>>>>>>>>>>>>Ro
>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>rt
>>>>>>>>>>>>R
>>>>>>>>>>>>ou
>>>>>>>>>>>>te
>>>>>>>>>>>>rs
>>>>>>>>>>>>(V
>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.dep
>>>>>>>>>>>>lo
>>>>>>>>>>>>y
>>>>>>>>>>>>Vi
>>>>>>>>>>>>rt
>>>>>>>>>>>>ua
>>>>>>>>>>>>lR
>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:190
>>>>>>>>>>>>2)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Virtua
>>>>>>>>>>>>lR
>>>>>>>>>>>>o
>>>>>>>>>>>>ut
>>>>>>>>>>>>er
>>>>>>>>>>>>El
>>>>>>>>>>>>em
>>>>>>>>>>>>ent.java:175)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAn
>>>>>>>>>>>>dR
>>>>>>>>>>>>e
>>>>>>>>>>>>so
>>>>>>>>>>>>ur
>>>>>>>>>>>>ce
>>>>>>>>>>>>s(
>>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMa
>>>>>>>>>>>>na
>>>>>>>>>>>>g
>>>>>>>>>>>>er
>>>>>>>>>>>>Im
>>>>>>>>>>>>pl
>>>>>>>>>>>>.j
>>>>>>>>>>>>ava:1434)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I
>>>>>>>>>>>>nt
>>>>>>>>>>>>e
>>>>>>>>>>>>rc
>>>>>>>>>>>>ep
>>>>>>>>>>>>to
>>>>>>>>>>>>rD
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12
>>>>>>>>>>>>5)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManage
>>>>>>>>>>>>rI
>>>>>>>>>>>>m
>>>>>>>>>>>>pl
>>>>>>>>>>>>.j
>>>>>>>>>>>>av
>>>>>>>>>>>>a:
>>>>>>>>>>>>2435)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>rt
>>>>>>>>>>>>R
>>>>>>>>>>>>ou
>>>>>>>>>>>>te
>>>>>>>>>>>>r(
>>>>>>>>>>>>Vi
>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta
>>>>>>>>>>>>rt
>>>>>>>>>>>>R
>>>>>>>>>>>>ou
>>>>>>>>>>>>te
>>>>>>>>>>>>r(
>>>>>>>>>>>>Vi
>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>>> at
>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I
>>>>>>>>>>>>nt
>>>>>>>>>>>>e
>>>>>>>>>>>>rc
>>>>>>>>>>>>ep
>>>>>>>>>>>>to
>>>>>>>>>>>>rD
>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12
>>>>>>>>>>>>5)
>>>>>>>>>>>> at
>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.e
>>>>>>>>>>>>xe
>>>>>>>>>>>>c
>>>>>>>>>>>>ut
>>>>>>>>>>>>e(
>>>>>>>>>>>>St
>>>>>>>>>>>>ar
>>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>table nics:
>>>>>>>>>>>>
>>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>>mac_address
>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>>>>> |
>>>>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>>>>> |
>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type
>>>>>>>>>>>>|
>>>>>>>>>>>>created
>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>>Ip4
>>>>>>>>>>>>|
>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>>0
>>>>>>>>>>>>|
>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>0 |
>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>|
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>>Ip4
>>>>>>>>>>>>|
>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>>0
>>>>>>>>>>>>|
>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL |
>>>>>>>>>>>>0 |
>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL
>>>>>>>>>>>>|
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL |
>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>|
>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>>> |
>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>|
>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>>> |
>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>|
>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>>> |
>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 |
>>>>>>>>>>>>NULL
>>>>>>>>>>>> | NULL | NULL | NULL | NULL |
>>>>>>>>>>>>NULL
>>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter
>>>>>>>>>>>>|
>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>>>>> |
>>>>>>>>>>>>+----+--------------------------------------+-------------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----+---------------+-------------+-------------+---------+----
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>+------------+--------+--------------+----------+--------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>-+
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------------------------------+-----------+----------------
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>+-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>----------+-------------+-------------+--------------------+---
>>>>>>>>>>>>--
>>>>>>>>>>>>-
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>--
>>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
select id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name from nics where instance_id=19;
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
| id | ip4_address | netmask | gateway | state | removed | network_id | reserver_name |
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL | 205 | ExternalGuestNetworkGuru |
| 30 | NULL | NULL | NULL | Allocated | NULL | 202 | ControlNetworkGuru |
| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL | 200 | PublicNetworkGuru |
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
and this is the nics element from the StartCmd. Is it normal to have
two nics with deviceId 0?
"nics":[
{"deviceId":2,
"networkRateMbps":200,
"defaultNic":true,
"uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
"ip":"10.193.17.190",
"netmask":"255.255.255.0",
"gateway":"10.193.17.1",
"mac":"06:7e:fe:00:00:bf",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Public",
"broadcastUri":"vlan://untagged",
"isolationUri":"vlan://untagged",
"isSecurityGroupEnabled":false,
"name":"cloudbr0"},
{"deviceId":0,
"networkRateMbps":200,
"defaultNic":false,
"uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
"ip":"10.124.99.1",
"netmask":"255.255.255.0",
"mac":"02:00:2a:43:00:0d",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Guest",
"broadcastUri":"vlan://3949",
"isolationUri":"vlan://3949",
"isSecurityGroupEnabled":false,
"name":"cloudbr1"},
{"deviceId":0,
"networkRateMbps":-1,
"defaultNic":false,
"uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
"ip":"169.254.1.59",
"netmask":"255.255.0.0",
"gateway":"169.254.0.1",
"mac":"0e:00:a9:fe:01:3b",
"broadcastType":"LinkLocal",
"type":"Control",
"isSecurityGroupEnabled":false}
]
Thanks
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Kambiz, the debug statements below are for the case when eth1 is a control
> interface as it was in your old command. I’ve looked at the new command,
> eth1 is not control, its either public or guest
>
> eth0: - control
>
> iface eth0 inet static
> address 169.254.1.59
> netmask 255.255.0.0
>
> eth1:
>
> iface eth1 inet static
> address 10.193.17.1
> Netmask
>
> So you need to execute the mysql statements for the traffic type of VR nic
> eth1
>
> -Alena.
>
>
>
>
> On 3/25/14, 9:57 AM, "Alena Prokharchyk" <Al...@citrix.com>
> wrote:
>
>>Kambiz, can you please check the following:
>>
>>
>>1) Check if the gateway is set on control network:
>>
>>mysql> select gateway, cidr from networks where traffic_type=‘Control’;
>>
>>2) For router control nic, check if network/gateway are set.
>>
>>Select gateway,netmask from nics where instance_id=<routerid> and
>>network_id=<control network id>
>>
>>-Alena.
>>
>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>I looked up the startup command of the old router instance which worked
>>>correctly:
>>>
>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga
>>>t
>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou
>>>d
>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%
>>>t
>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>
>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>eth1mask.
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>eth1
>>>> doesn’t seem related to the problem (although we have to figure out why
>>>> its missing), as the connection that fails, happening to link local
>>>>169.x
>>>> eth0 interface.
>>>>
>>>> Edison, can you please tell us how to debug link local connection
>>>>failure,
>>>> on KVM agent?
>>>>
>>>> Thank you,
>>>> Alena.
>>>>
>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>thank you, the NullPointerException doesn't occur any more, but there
>>>>>still seems to be a problem during startup of the router.
>>>>>
>>>>>When I start the virtual router, it comes up, but in agent.log, there
>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>
>>>>>Then I logged into the virtual router to find out that the netmask of
>>>>>eth1 is missing in the interfaces file:
>>>>>
>>>>>root@host:~# virsh console r-19-VM
>>>>>Connected to domain r-19-VM
>>>>>Escape character is ^]
>>>>>
>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>
>>>>>r-19-VM login: root
>>>>>...
>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>auto lo eth0 eth1 eth2
>>>>>iface lo inet loopback
>>>>>
>>>>>iface eth0 inet static
>>>>> address 169.254.1.59
>>>>> netmask 255.255.0.0
>>>>>iface eth1 inet static
>>>>> address 10.193.17.1
>>>>> netmask
>>>>>iface eth2 inet static
>>>>> address 10.193.17.190
>>>>> netmask 255.255.255.0
>>>>>
>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>where the parameters are visible:
>>>>>
>>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0
>>>>>%
>>>>>ga
>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6c
>>>>>l
>>>>>ou
>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0
>>>>>.
>>>>>0%
>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>
>>>>>
>>>>>Any hint is appreciated.
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If
>>>>>>vm
>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>> user_ip_address table. Here is the flow:
>>>>>>
>>>>>> 1) Save the db dump.
>>>>>> 2) Run the query to cleanup the reference:
>>>>>>
>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>>>> id=<problematic public ip address id>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Let me know how it works.
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>query
>>>>>>>correctly:
>>>>>>>
>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>from user_ip_address uip
>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>>>fr.network_id=205);
>>>>>>>
>>>>>>>
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>>>>>> | vm_state | vm_removed |
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>
>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>
>>>>>>>Thank you
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>
>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>right?)
>>>>>>>>
>>>>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>>>>
>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>
>>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>>>id,
>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>
>>>>>>>> For each vmId/networkId combo, check if there is non-removed nic
>>>>>>>>and
>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>reference
>>>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>>>know
>>>>>>>>and
>>>>>>>> I will tell you how to clean it up
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hi Alena,
>>>>>>>>>
>>>>>>>>>thank you for your help.
>>>>>>>>>
>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>>>removed
>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>created
>>>>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>>>>change by restoring from the dump.
>>>>>>>>>
>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>>>>which
>>>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>>>the
>>>>>>>>>first
>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>suggested
>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>procedure
>>>>>>>>>solved the problem).
>>>>>>>>>
>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>
>>>>>>>>>Anything else I can try?
>>>>>>>>>
>>>>>>>>>Thank you
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>i.name,
>>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>>>i.id
>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>gateway
>>>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>>>name
>>>>>>>>>| state | type |
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>
>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> The error happens not because Ip is null, but because the nic in
>>>>>>>>>>a
>>>>>>>>>>certain
>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>
>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>
>>>>>>>>>> 1) Stop the MS
>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>>>>>> 3) Run the query:
>>>>>>>>>>
>>>>>>>>>> select * from router_network_ref where router_id=<id of your VR)
>>>>>>>>>>and
>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>instance_id=<ID
>>>>>>>>>>of
>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>
>>>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>>>cleaned
>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>router_network_ref
>>>>>>>>>> table.
>>>>>>>>>>
>>>>>>>>>> Let me know how it works.
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hello,
>>>>>>>>>>>
>>>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>>>any
>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>
>>>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>>>the
>>>>>>>>>>>java
>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>
>>>>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>>>>out
>>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>>>(cf.
>>>>>>>>>>>row
>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>
>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>suggestions
>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>
>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>management-server.log:
>>>>>>>>>>>
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>>for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>associated
>>>>>>>>>>>with
>>>>>>>>>>>any VPC
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>>service
>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported
>>>>>>>>>>>in
>>>>>>>>>>>the
>>>>>>>>>>>network id=205
>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>>>>part
>>>>>>>>>>>of
>>>>>>>>>>>network implement
>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>>>>part
>>>>>>>>>>>of
>>>>>>>>>>>network implement
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>>for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>associated
>>>>>>>>>>>with any VPC
>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>>service
>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>>>>for
>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>>>adding
>>>>>>>>>>>disk to VM
>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>apply
>>>>>>>>>>>as
>>>>>>>>>>>a
>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImp
>>>>>>>>>>>l
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:
>>>>>>>>>>>76
>>>>>>>>>>>3)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>Ne
>>>>>>>>>>>tw
>>>>>>>>>>>or
>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>>n
>>>>>>>>>>>al
>>>>>>>>>>>iz
>>>>>>>>>>>eN
>>>>>>>>>>>et
>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:92
>>>>>>>>>>>8
>>>>>>>>>>>)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>Co
>>>>>>>>>>>mm
>>>>>>>>>>>an
>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>>n
>>>>>>>>>>>al
>>>>>>>>>>>iz
>>>>>>>>>>>eC
>>>>>>>>>>>om
>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>De
>>>>>>>>>>>pl
>>>>>>>>>>>oy
>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachin
>>>>>>>>>>>e
>>>>>>>>>>>Ma
>>>>>>>>>>>na
>>>>>>>>>>>ge
>>>>>>>>>>>rI
>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManage
>>>>>>>>>>>r
>>>>>>>>>>>Im
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:471)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>(
>>>>>>>>>>>Vi
>>>>>>>>>>>rt
>>>>>>>>>>>ua
>>>>>>>>>>>lN
>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>V
>>>>>>>>>>>ir
>>>>>>>>>>>tu
>>>>>>>>>>>al
>>>>>>>>>>>Ro
>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>rs
>>>>>>>>>>>(V
>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deplo
>>>>>>>>>>>y
>>>>>>>>>>>Vi
>>>>>>>>>>>rt
>>>>>>>>>>>ua
>>>>>>>>>>>lR
>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualR
>>>>>>>>>>>o
>>>>>>>>>>>ut
>>>>>>>>>>>er
>>>>>>>>>>>El
>>>>>>>>>>>em
>>>>>>>>>>>ent.java:175)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndR
>>>>>>>>>>>e
>>>>>>>>>>>so
>>>>>>>>>>>ur
>>>>>>>>>>>ce
>>>>>>>>>>>s(
>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMana
>>>>>>>>>>>g
>>>>>>>>>>>er
>>>>>>>>>>>Im
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>ava:1434)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>>e
>>>>>>>>>>>rc
>>>>>>>>>>>ep
>>>>>>>>>>>to
>>>>>>>>>>>rD
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerI
>>>>>>>>>>>m
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:
>>>>>>>>>>>2435)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>r(
>>>>>>>>>>>Vi
>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>r(
>>>>>>>>>>>Vi
>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>>e
>>>>>>>>>>>rc
>>>>>>>>>>>ep
>>>>>>>>>>>to
>>>>>>>>>>>rD
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>> at
>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exe
>>>>>>>>>>>c
>>>>>>>>>>>ut
>>>>>>>>>>>e(
>>>>>>>>>>>St
>>>>>>>>>>>ar
>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>table nics:
>>>>>>>>>>>
>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>mac_address
>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>>>> |
>>>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>>>> |
>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>>>>>created
>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>Ip4
>>>>>>>>>>>|
>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>0
>>>>>>>>>>>|
>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>>NULL
>>>>>>>>>>> | NULL |
>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>Ip4
>>>>>>>>>>>|
>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>0
>>>>>>>>>>>|
>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>>NULL
>>>>>>>>>>> | NULL |
>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>
>>>>>>>>>>>
>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Kambiz, the debug statements below are for the case when eth1 is a control
interface as it was in your old command. I’ve looked at the new command,
eth1 is not control, its either public or guest
eth0: - control
iface eth0 inet static
address 169.254.1.59
netmask 255.255.0.0
eth1:
iface eth1 inet static
address 10.193.17.1
Netmask
So you need to execute the mysql statements for the traffic type of VR nic
eth1
-Alena.
On 3/25/14, 9:57 AM, "Alena Prokharchyk" <Al...@citrix.com>
wrote:
>Kambiz, can you please check the following:
>
>
>1) Check if the gateway is set on control network:
>
>mysql> select gateway, cidr from networks where traffic_type=‘Control’;
>
>2) For router control nic, check if network/gateway are set.
>
>Select gateway,netmask from nics where instance_id=<routerid> and
>network_id=<control network id>
>
>-Alena.
>
>On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>I looked up the startup command of the old router instance which worked
>>correctly:
>>
>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga
>>t
>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou
>>d
>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%
>>t
>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>
>>The new command (cf. below) doesn't have the parameters eth1ip and
>>eth1mask.
>>
>>Thanks
>>
>>
>>Kambiz
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>>> static nat rule programming for non-existing vm. Missing netmask on
>>>eth1
>>> doesn’t seem related to the problem (although we have to figure out why
>>> its missing), as the connection that fails, happening to link local
>>>169.x
>>> eth0 interface.
>>>
>>> Edison, can you please tell us how to debug link local connection
>>>failure,
>>> on KVM agent?
>>>
>>> Thank you,
>>> Alena.
>>>
>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>thank you, the NullPointerException doesn't occur any more, but there
>>>>still seems to be a problem during startup of the router.
>>>>
>>>>When I start the virtual router, it comes up, but in agent.log, there
>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>
>>>>Then I logged into the virtual router to find out that the netmask of
>>>>eth1 is missing in the interfaces file:
>>>>
>>>>root@host:~# virsh console r-19-VM
>>>>Connected to domain r-19-VM
>>>>Escape character is ^]
>>>>
>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>
>>>>r-19-VM login: root
>>>>...
>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>auto lo eth0 eth1 eth2
>>>>iface lo inet loopback
>>>>
>>>>iface eth0 inet static
>>>> address 169.254.1.59
>>>> netmask 255.255.0.0
>>>>iface eth1 inet static
>>>> address 10.193.17.1
>>>> netmask
>>>>iface eth2 inet static
>>>> address 10.193.17.190
>>>> netmask 255.255.255.0
>>>>
>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>where the parameters are visible:
>>>>
>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>>(agentRequest-Handler-2:null) Executing:
>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0
>>>>%
>>>>ga
>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6c
>>>>l
>>>>ou
>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0
>>>>.
>>>>0%
>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>
>>>>
>>>>Any hint is appreciated.
>>>>
>>>>Thanks
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If
>>>>>vm
>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>> user_ip_address table. Here is the flow:
>>>>>
>>>>> 1) Save the db dump.
>>>>> 2) Run the query to cleanup the reference:
>>>>>
>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>>> id=<problematic public ip address id>
>>>>>
>>>>>
>>>>>
>>>>> Let me know how it works.
>>>>>
>>>>> -Alena.
>>>>>
>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>I hope I have understood what you wrote and created the following
>>>>>>query
>>>>>>correctly:
>>>>>>
>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>from user_ip_address uip
>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>>fr.network_id=205);
>>>>>>
>>>>>>
>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>-
>>>>>>--
>>>>>>--
>>>>>>----+-----------+------------+
>>>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>>>>> | vm_state | vm_removed |
>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>-
>>>>>>--
>>>>>>--
>>>>>>----+-----------+------------+
>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>>>>23:00:53 | Expunging | NULL |
>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>> | Stopped | NULL |
>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>-
>>>>>>--
>>>>>>--
>>>>>>----+-----------+------------+
>>>>>>
>>>>>>Is VM id 15 what you are looking for?
>>>>>>
>>>>>>Thank you
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>
>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>right?)
>>>>>>>
>>>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>>>
>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>
>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>>id,
>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>
>>>>>>> For each vmId/networkId combo, check if there is non-removed nic
>>>>>>>and
>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>reference
>>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>>know
>>>>>>>and
>>>>>>> I will tell you how to clean it up
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>>>Hi Alena,
>>>>>>>>
>>>>>>>>thank you for your help.
>>>>>>>>
>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>>removed
>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>created
>>>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>>>change by restoring from the dump.
>>>>>>>>
>>>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>>>which
>>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>>the
>>>>>>>>first
>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>suggested
>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>procedure
>>>>>>>>solved the problem).
>>>>>>>>
>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>
>>>>>>>>Anything else I can try?
>>>>>>>>
>>>>>>>>Thank you
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>i.name,
>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>>i.id
>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>-----------+--------------+
>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>gateway
>>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>>name
>>>>>>>>| state | type |
>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>-----------+--------------+
>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>>r-4-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>>r-4-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>10.193.17.1
>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>>r-4-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>>r-7-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>>r-7-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>10.193.17.1
>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>>r-7-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>>r-18-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>>r-18-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>10.193.17.1
>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>>r-18-VM
>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>>r-19-VM
>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>>r-19-VM
>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>10.193.17.1
>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>>r-19-VM
>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-+
>>>>>>>>-----------+--------------+
>>>>>>>>
>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> The error happens not because Ip is null, but because the nic in
>>>>>>>>>a
>>>>>>>>>certain
>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>
>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>
>>>>>>>>> 1) Stop the MS
>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>>>>> 3) Run the query:
>>>>>>>>>
>>>>>>>>> select * from router_network_ref where router_id=<id of your VR)
>>>>>>>>>and
>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>instance_id=<ID
>>>>>>>>>of
>>>>>>>>> your VR> and removed is null);
>>>>>>>>>
>>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>>cleaned
>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>router_network_ref
>>>>>>>>> table.
>>>>>>>>>
>>>>>>>>> Let me know how it works.
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hello,
>>>>>>>>>>
>>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>>any
>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>
>>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>>the
>>>>>>>>>>java
>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>
>>>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>>>out
>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>>(cf.
>>>>>>>>>>row
>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>
>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>suggestions
>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>
>>>>>>>>>>Thank you for your help
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Kambiz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>management-server.log:
>>>>>>>>>>
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>for
>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>associated
>>>>>>>>>>with
>>>>>>>>>>any VPC
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>service
>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported
>>>>>>>>>>in
>>>>>>>>>>the
>>>>>>>>>>network id=205
>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>>>part
>>>>>>>>>>of
>>>>>>>>>>network implement
>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>>>part
>>>>>>>>>>of
>>>>>>>>>>network implement
>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>for
>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>associated
>>>>>>>>>>with any VPC
>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>service
>>>>>>>>>>Connectivity on network null
>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>>>for
>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>>adding
>>>>>>>>>>disk to VM
>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>apply
>>>>>>>>>>as
>>>>>>>>>>a
>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImp
>>>>>>>>>>l
>>>>>>>>>>.j
>>>>>>>>>>av
>>>>>>>>>>a:
>>>>>>>>>>76
>>>>>>>>>>3)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>i
>>>>>>>>>>ze
>>>>>>>>>>Ne
>>>>>>>>>>tw
>>>>>>>>>>or
>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>n
>>>>>>>>>>al
>>>>>>>>>>iz
>>>>>>>>>>eN
>>>>>>>>>>et
>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:92
>>>>>>>>>>8
>>>>>>>>>>)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>i
>>>>>>>>>>ze
>>>>>>>>>>Co
>>>>>>>>>>mm
>>>>>>>>>>an
>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>n
>>>>>>>>>>al
>>>>>>>>>>iz
>>>>>>>>>>eC
>>>>>>>>>>om
>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>i
>>>>>>>>>>ze
>>>>>>>>>>De
>>>>>>>>>>pl
>>>>>>>>>>oy
>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachin
>>>>>>>>>>e
>>>>>>>>>>Ma
>>>>>>>>>>na
>>>>>>>>>>ge
>>>>>>>>>>rI
>>>>>>>>>>mpl.java:763)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManage
>>>>>>>>>>r
>>>>>>>>>>Im
>>>>>>>>>>pl
>>>>>>>>>>.j
>>>>>>>>>>av
>>>>>>>>>>a:471)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>(
>>>>>>>>>>Vi
>>>>>>>>>>rt
>>>>>>>>>>ua
>>>>>>>>>>lN
>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>V
>>>>>>>>>>ir
>>>>>>>>>>tu
>>>>>>>>>>al
>>>>>>>>>>Ro
>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>R
>>>>>>>>>>ou
>>>>>>>>>>te
>>>>>>>>>>rs
>>>>>>>>>>(V
>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deplo
>>>>>>>>>>y
>>>>>>>>>>Vi
>>>>>>>>>>rt
>>>>>>>>>>ua
>>>>>>>>>>lR
>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualR
>>>>>>>>>>o
>>>>>>>>>>ut
>>>>>>>>>>er
>>>>>>>>>>El
>>>>>>>>>>em
>>>>>>>>>>ent.java:175)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndR
>>>>>>>>>>e
>>>>>>>>>>so
>>>>>>>>>>ur
>>>>>>>>>>ce
>>>>>>>>>>s(
>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMana
>>>>>>>>>>g
>>>>>>>>>>er
>>>>>>>>>>Im
>>>>>>>>>>pl
>>>>>>>>>>.j
>>>>>>>>>>ava:1434)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>e
>>>>>>>>>>rc
>>>>>>>>>>ep
>>>>>>>>>>to
>>>>>>>>>>rD
>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerI
>>>>>>>>>>m
>>>>>>>>>>pl
>>>>>>>>>>.j
>>>>>>>>>>av
>>>>>>>>>>a:
>>>>>>>>>>2435)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>R
>>>>>>>>>>ou
>>>>>>>>>>te
>>>>>>>>>>r(
>>>>>>>>>>Vi
>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>R
>>>>>>>>>>ou
>>>>>>>>>>te
>>>>>>>>>>r(
>>>>>>>>>>Vi
>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>> at
>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>e
>>>>>>>>>>rc
>>>>>>>>>>ep
>>>>>>>>>>to
>>>>>>>>>>rD
>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>> at
>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exe
>>>>>>>>>>c
>>>>>>>>>>ut
>>>>>>>>>>e(
>>>>>>>>>>St
>>>>>>>>>>ar
>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>table nics:
>>>>>>>>>>
>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>+-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>mac_address
>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>broadcast_uri
>>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>>> |
>>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>>> |
>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>>>>created
>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>+-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>Ip4
>>>>>>>>>>|
>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>0
>>>>>>>>>>|
>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>NULL
>>>>>>>>>> | NULL |
>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>Ip4
>>>>>>>>>>|
>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>0
>>>>>>>>>>|
>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>NULL
>>>>>>>>>> | NULL |
>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>12:27:58 |
>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>> |
>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>12:27:52 |
>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>> |
>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>09:11:16 |
>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>> |
>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>09:11:44 |
>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>>> |
>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>-+
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>+-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>-
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>--
>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>
>>>>>>>>>>
>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Kambiz, can you please check the following:
1) Check if the gateway is set on control network:
mysql> select gateway, cidr from networks where traffic_type=‘Control’;
2) For router control nic, check if network/gateway are set.
Select gateway,netmask from nics where instance_id=<routerid> and
network_id=<control network id>
-Alena.
On 3/25/14, 5:47 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>I looked up the startup command of the old router instance which worked
>correctly:
>
>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gat
>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud
>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%t
>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>
>The new command (cf. below) doesn't have the parameters eth1ip and
>eth1mask.
>
>Thanks
>
>
>Kambiz
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>> static nat rule programming for non-existing vm. Missing netmask on eth1
>> doesn’t seem related to the problem (although we have to figure out why
>> its missing), as the connection that fails, happening to link local
>>169.x
>> eth0 interface.
>>
>> Edison, can you please tell us how to debug link local connection
>>failure,
>> on KVM agent?
>>
>> Thank you,
>> Alena.
>>
>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>thank you, the NullPointerException doesn't occur any more, but there
>>>still seems to be a problem during startup of the router.
>>>
>>>When I start the virtual router, it comes up, but in agent.log, there
>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>
>>>Then I logged into the virtual router to find out that the netmask of
>>>eth1 is missing in the interfaces file:
>>>
>>>root@host:~# virsh console r-19-VM
>>>Connected to domain r-19-VM
>>>Escape character is ^]
>>>
>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>
>>>r-19-VM login: root
>>>...
>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>auto lo eth0 eth1 eth2
>>>iface lo inet loopback
>>>
>>>iface eth0 inet static
>>> address 169.254.1.59
>>> netmask 255.255.0.0
>>>iface eth1 inet static
>>> address 10.193.17.1
>>> netmask
>>>iface eth2 inet static
>>> address 10.193.17.190
>>> netmask 255.255.255.0
>>>
>>>I don't know if it is relevant, but this is the line from agent.log
>>>where the parameters are visible:
>>>
>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>(agentRequest-Handler-2:null) Executing:
>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%
>>>ga
>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl
>>>ou
>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.
>>>0%
>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>
>>>
>>>Any hint is appreciated.
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
>>>> id=15 is expunged, we have to clear out the reference to it from
>>>> user_ip_address table. Here is the flow:
>>>>
>>>> 1) Save the db dump.
>>>> 2) Run the query to cleanup the reference:
>>>>
>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>> id=<problematic public ip address id>
>>>>
>>>>
>>>>
>>>> Let me know how it works.
>>>>
>>>> -Alena.
>>>>
>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>I hope I have understood what you wrote and created the following
>>>>>query
>>>>>correctly:
>>>>>
>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>from user_ip_address uip
>>>>> join nics n on uip.vm_id = n.instance_id
>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>fr.network_id=205);
>>>>>
>>>>>
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>>>> | vm_state | vm_removed |
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>>>23:00:53 | Expunging | NULL |
>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>> | Stopped | NULL |
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>
>>>>>Is VM id 15 what you are looking for?
>>>>>
>>>>>Thank you
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> Kambiz, can you please try one more thing.
>>>>>>
>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>right?)
>>>>>>
>>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>>
>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>
>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>id,
>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>
>>>>>> For each vmId/networkId combo, check if there is non-removed nic and
>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>reference
>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>know
>>>>>>and
>>>>>> I will tell you how to clean it up
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi Alena,
>>>>>>>
>>>>>>>thank you for your help.
>>>>>>>
>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>removed
>>>>>>>the row though to see what happens: a new virtual router was created
>>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>>change by restoring from the dump.
>>>>>>>
>>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>>which
>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>the
>>>>>>>first
>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>suggested
>>>>>>>that the deleted router would be recreated again (and this procedure
>>>>>>>solved the problem).
>>>>>>>
>>>>>>>Below I have attached the state of the two tables.
>>>>>>>
>>>>>>>Anything else I can try?
>>>>>>>
>>>>>>>Thank you
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>i.name,
>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>i.id
>>>>>>>where i.type = 'DomainRouter';
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>| id | removed | ip4_address | netmask | gateway
>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>name
>>>>>>>| state | type |
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped | DomainRouter |
>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped | DomainRouter |
>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped | DomainRouter |
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>
>>>>>>>mysql> select * from router_network_ref;
>>>>>>>+----+-----------+------------+------------+
>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>+----+-----------+------------+------------+
>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>+----+-----------+------------+------------+
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> The error happens not because Ip is null, but because the nic in a
>>>>>>>>certain
>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>
>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>
>>>>>>>> 1) Stop the MS
>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>>>> 3) Run the query:
>>>>>>>>
>>>>>>>> select * from router_network_ref where router_id=<id of your VR)
>>>>>>>>and
>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>instance_id=<ID
>>>>>>>>of
>>>>>>>> your VR> and removed is null);
>>>>>>>>
>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>cleaned
>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>router_network_ref
>>>>>>>> table.
>>>>>>>>
>>>>>>>> Let me know how it works.
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hello,
>>>>>>>>>
>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>any
>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>
>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>the
>>>>>>>>>java
>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>
>>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>>out
>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>(cf.
>>>>>>>>>row
>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>
>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>suggestions
>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>
>>>>>>>>>Thank you for your help
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>management-server.log:
>>>>>>>>>
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>associated
>>>>>>>>>with
>>>>>>>>>any VPC
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>service
>>>>>>>>>Connectivity on network net1
>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in
>>>>>>>>>the
>>>>>>>>>network id=205
>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>>part
>>>>>>>>>of
>>>>>>>>>network implement
>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>>part
>>>>>>>>>of
>>>>>>>>>network implement
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>associated
>>>>>>>>>with any VPC
>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>service
>>>>>>>>>Connectivity on network null
>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>>for
>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>adding
>>>>>>>>>disk to VM
>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply
>>>>>>>>>as
>>>>>>>>>a
>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>java.lang.NullPointerException
>>>>>>>>> at
>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:
>>>>>>>>>76
>>>>>>>>>3)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>Ne
>>>>>>>>>tw
>>>>>>>>>or
>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>al
>>>>>>>>>iz
>>>>>>>>>eN
>>>>>>>>>et
>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928
>>>>>>>>>)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>Co
>>>>>>>>>mm
>>>>>>>>>an
>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>al
>>>>>>>>>iz
>>>>>>>>>eC
>>>>>>>>>om
>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>De
>>>>>>>>>pl
>>>>>>>>>oy
>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>> at
>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachine
>>>>>>>>>Ma
>>>>>>>>>na
>>>>>>>>>ge
>>>>>>>>>rI
>>>>>>>>>mpl.java:763)
>>>>>>>>> at
>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManager
>>>>>>>>>Im
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:471)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(
>>>>>>>>>Vi
>>>>>>>>>rt
>>>>>>>>>ua
>>>>>>>>>lN
>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startV
>>>>>>>>>ir
>>>>>>>>>tu
>>>>>>>>>al
>>>>>>>>>Ro
>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>rs
>>>>>>>>>(V
>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deploy
>>>>>>>>>Vi
>>>>>>>>>rt
>>>>>>>>>ua
>>>>>>>>>lR
>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRo
>>>>>>>>>ut
>>>>>>>>>er
>>>>>>>>>El
>>>>>>>>>em
>>>>>>>>>ent.java:175)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndRe
>>>>>>>>>so
>>>>>>>>>ur
>>>>>>>>>ce
>>>>>>>>>s(
>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManag
>>>>>>>>>er
>>>>>>>>>Im
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>ava:1434)
>>>>>>>>> at
>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte
>>>>>>>>>rc
>>>>>>>>>ep
>>>>>>>>>to
>>>>>>>>>rD
>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerIm
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:
>>>>>>>>>2435)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>r(
>>>>>>>>>Vi
>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>> at
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>r(
>>>>>>>>>Vi
>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>> at
>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte
>>>>>>>>>rc
>>>>>>>>>ep
>>>>>>>>>to
>>>>>>>>>rD
>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>> at
>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exec
>>>>>>>>>ut
>>>>>>>>>e(
>>>>>>>>>St
>>>>>>>>>ar
>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>table nics:
>>>>>>>>>
>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>mac_address
>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>broadcast_uri
>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>> |
>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>> |
>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>>>created
>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>Ip4
>>>>>>>>>|
>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>0
>>>>>>>>>|
>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>NULL
>>>>>>>>> | NULL |
>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>Ip4
>>>>>>>>>|
>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>0
>>>>>>>>>|
>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>NULL
>>>>>>>>> | NULL |
>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>12:27:58 |
>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>> |
>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>12:27:52 |
>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>> |
>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>09:11:16 |
>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>> |
>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>09:11:44 |
>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>> |
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>
>>>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
I looked up the startup command of the old router instance which worked
correctly:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p %template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gateway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=10.193.17.1
The new command (cf. below) doesn't have the parameters eth1ip and
eth1mask.
Thanks
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> I don’t think its relevant as the piece we’ve fixed, just eliminated
> static nat rule programming for non-existing vm. Missing netmask on eth1
> doesn’t seem related to the problem (although we have to figure out why
> its missing), as the connection that fails, happening to link local 169.x
> eth0 interface.
>
> Edison, can you please tell us how to debug link local connection failure,
> on KVM agent?
>
> Thank you,
> Alena.
>
> On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>thank you, the NullPointerException doesn't occur any more, but there
>>still seems to be a problem during startup of the router.
>>
>>When I start the virtual router, it comes up, but in agent.log, there
>>are lots of 'Could not connect to 169.254.1.x' messages.
>>
>>Then I logged into the virtual router to find out that the netmask of
>>eth1 is missing in the interfaces file:
>>
>>root@host:~# virsh console r-19-VM
>>Connected to domain r-19-VM
>>Escape character is ^]
>>
>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>
>>r-19-VM login: root
>>...
>>root@r-19-VM:~# cat /etc/network/interfaces
>>auto lo eth0 eth1 eth2
>>iface lo inet loopback
>>
>>iface eth0 inet static
>> address 169.254.1.59
>> netmask 255.255.0.0
>>iface eth1 inet static
>> address 10.193.17.1
>> netmask
>>iface eth2 inet static
>> address 10.193.17.190
>> netmask 255.255.255.0
>>
>>I don't know if it is relevant, but this is the line from agent.log
>>where the parameters are visible:
>>
>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>(agentRequest-Handler-2:null) Executing:
>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga
>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou
>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.0%
>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>
>>
>>Any hint is appreciated.
>>
>>Thanks
>>
>>
>>Kambiz
>>
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
>>> id=15 is expunged, we have to clear out the reference to it from
>>> user_ip_address table. Here is the flow:
>>>
>>> 1) Save the db dump.
>>> 2) Run the query to cleanup the reference:
>>>
>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>> id=<problematic public ip address id>
>>>
>>>
>>>
>>> Let me know how it works.
>>>
>>> -Alena.
>>>
>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>I hope I have understood what you wrote and created the following query
>>>>correctly:
>>>>
>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>> n.state as nic_state, n.removed as nic_removed,
>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>from user_ip_address uip
>>>> join nics n on uip.vm_id = n.instance_id
>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>fr.network_id=205);
>>>>
>>>>
>>>>+-------+------------+-------------------+--------------+---------------
>>>>--
>>>>----+-----------+------------+
>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>>> | vm_state | vm_removed |
>>>>+-------+------------+-------------------+--------------+---------------
>>>>--
>>>>----+-----------+------------+
>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>>23:00:53 | Expunging | NULL |
>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>> | Stopped | NULL |
>>>>+-------+------------+-------------------+--------------+---------------
>>>>--
>>>>----+-----------+------------+
>>>>
>>>>Is VM id 15 what you are looking for?
>>>>
>>>>Thank you
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> Kambiz, can you please try one more thing.
>>>>>
>>>>> 1) Locate all the firewall rules for your guest network (205, right?)
>>>>>
>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>
>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>
>>>>> Select vm_id, network_id from user_ip_address where id in (Select id,
>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>
>>>>> For each vmId/networkId combo, check if there is non-removed nic and
>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>reference
>>>>> referring to vm that is removed already. If you find any, let me know
>>>>>and
>>>>> I will tell you how to clean it up
>>>>>
>>>>> -Alena.
>>>>>
>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hi Alena,
>>>>>>
>>>>>>thank you for your help.
>>>>>>
>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>removed
>>>>>>the row though to see what happens: a new virtual router was created
>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>change by restoring from the dump.
>>>>>>
>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>which
>>>>>>was used by my instances. I deleted the router using the UI after the
>>>>>>first
>>>>>>occurrence of the NPE, because a post with a similar problem suggested
>>>>>>that the deleted router would be recreated again (and this procedure
>>>>>>solved the problem).
>>>>>>
>>>>>>Below I have attached the state of the two tables.
>>>>>>
>>>>>>Anything else I can try?
>>>>>>
>>>>>>Thank you
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>>>>>>where i.type = 'DomainRouter';
>>>>>>+----+---------------------+---------------+---------------+----------
>>>>>>--
>>>>>>-+
>>>>>>---------+--------------------------+------------+-------------+------
>>>>>>--
>>>>>>-+
>>>>>>-----------+--------------+
>>>>>>| id | removed | ip4_address | netmask | gateway
>>>>>>| ip_type | reserver_name | network_id | instance_id | name
>>>>>>| state | type |
>>>>>>+----+---------------------+---------------+---------------+----------
>>>>>>--
>>>>>>-+
>>>>>>---------+--------------------------+------------+-------------+------
>>>>>>--
>>>>>>-+
>>>>>>-----------+--------------+
>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>r-4-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>r-4-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>10.193.17.1
>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>r-4-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>r-7-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>r-7-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>10.193.17.1
>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>r-7-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>r-18-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>r-18-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>10.193.17.1
>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>r-18-VM
>>>>>>| Expunging | DomainRouter |
>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>r-19-VM
>>>>>>| Stopped | DomainRouter |
>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>r-19-VM
>>>>>>| Stopped | DomainRouter |
>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>10.193.17.1
>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>r-19-VM
>>>>>>| Stopped | DomainRouter |
>>>>>>+----+---------------------+---------------+---------------+----------
>>>>>>--
>>>>>>-+
>>>>>>---------+--------------------------+------------+-------------+------
>>>>>>--
>>>>>>-+
>>>>>>-----------+--------------+
>>>>>>
>>>>>>mysql> select * from router_network_ref;
>>>>>>+----+-----------+------------+------------+
>>>>>>| id | router_id | network_id | guest_type |
>>>>>>+----+-----------+------------+------------+
>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>+----+-----------+------------+------------+
>>>>>>
>>>>>>
>>>>>>
>>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>>
>>>>>>> The error happens not because Ip is null, but because the nic in a
>>>>>>>certain
>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>> plug/unplug for Guest networks process.
>>>>>>>
>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>
>>>>>>> 1) Stop the MS
>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>>> 3) Run the query:
>>>>>>>
>>>>>>> select * from router_network_ref where router_id=<id of your VR) and
>>>>>>> network_id not in (select network_id from nics where instance_id=<ID
>>>>>>>of
>>>>>>> your VR> and removed is null);
>>>>>>>
>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>cleaned
>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>router_network_ref
>>>>>>> table.
>>>>>>>
>>>>>>> Let me know how it works.
>>>>>>>
>>>>>>> -Alena.
>>>>>>>
>>>>>>>
>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>>
>>>>>>>>Hello,
>>>>>>>>
>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>contributors for Cloudstack which I use since last fall without any
>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>
>>>>>>>>After a restart of the management server (stopping and starting the
>>>>>>>>java
>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>
>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>out
>>>>>>>>that the reason is a row in the table nics which has NULL in ip (cf.
>>>>>>>>row
>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>
>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>suggestions
>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>
>>>>>>>>Thank you for your help
>>>>>>>>
>>>>>>>>
>>>>>>>>Kambiz
>>>>>>>>
>>>>>>>>
>>>>>>>>management-server.log:
>>>>>>>>
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>>>>>>with
>>>>>>>>any VPC
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>service
>>>>>>>>Connectivity on network net1
>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in
>>>>>>>>the
>>>>>>>>network id=205
>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>part
>>>>>>>>of
>>>>>>>>network implement
>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>part
>>>>>>>>of
>>>>>>>>network implement
>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>associated
>>>>>>>>with any VPC
>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>service
>>>>>>>>Connectivity on network null
>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>for
>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>adding
>>>>>>>>disk to VM
>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>dns1=10.193.17.1
>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply
>>>>>>>>as
>>>>>>>>a
>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>java.lang.NullPointerException
>>>>>>>> at
>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.j
>>>>>>>>av
>>>>>>>>a:
>>>>>>>>76
>>>>>>>>3)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>>Ne
>>>>>>>>tw
>>>>>>>>or
>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final
>>>>>>>>iz
>>>>>>>>eN
>>>>>>>>et
>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>>Co
>>>>>>>>mm
>>>>>>>>an
>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final
>>>>>>>>iz
>>>>>>>>eC
>>>>>>>>om
>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>>De
>>>>>>>>pl
>>>>>>>>oy
>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>> at
>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMa
>>>>>>>>na
>>>>>>>>ge
>>>>>>>>rI
>>>>>>>>mpl.java:763)
>>>>>>>> at
>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm
>>>>>>>>pl
>>>>>>>>.j
>>>>>>>>av
>>>>>>>>a:471)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Vi
>>>>>>>>rt
>>>>>>>>ua
>>>>>>>>lN
>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVir
>>>>>>>>tu
>>>>>>>>al
>>>>>>>>Ro
>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>>te
>>>>>>>>rs
>>>>>>>>(V
>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVi
>>>>>>>>rt
>>>>>>>>ua
>>>>>>>>lR
>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>> at
>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRout
>>>>>>>>er
>>>>>>>>El
>>>>>>>>em
>>>>>>>>ent.java:175)
>>>>>>>> at
>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndReso
>>>>>>>>ur
>>>>>>>>ce
>>>>>>>>s(
>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>> at
>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManager
>>>>>>>>Im
>>>>>>>>pl
>>>>>>>>.j
>>>>>>>>ava:1434)
>>>>>>>> at
>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>>ep
>>>>>>>>to
>>>>>>>>rD
>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>> at
>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl
>>>>>>>>.j
>>>>>>>>av
>>>>>>>>a:
>>>>>>>>2435)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>>te
>>>>>>>>r(
>>>>>>>>Vi
>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>> at
>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>>te
>>>>>>>>r(
>>>>>>>>Vi
>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>> at
>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>>ep
>>>>>>>>to
>>>>>>>>rD
>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>> at
>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execut
>>>>>>>>e(
>>>>>>>>St
>>>>>>>>ar
>>>>>>>>tRouterCmd.java:103)
>>>>>>>>
>>>>>>>>
>>>>>>>>table nics:
>>>>>>>>
>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>'ControlNetworkGuru';
>>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>>-+
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>>+-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>| id | uuid | instance_id |
>>>>>>>>mac_address
>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>broadcast_uri
>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>> |
>>>>>>>>reservation_id | device_id | update_time
>>>>>>>> |
>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>>created
>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>>-+
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>>+-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>>>|
>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>0
>>>>>>>>|
>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>>NULL
>>>>>>>> | NULL |
>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>>>|
>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>0
>>>>>>>>|
>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>>NULL
>>>>>>>> | NULL |
>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>ControlNetworkGuru |
>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>12:27:58 |
>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>ControlNetworkGuru |
>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>12:27:52 |
>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>ControlNetworkGuru |
>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>09:11:16 |
>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>ControlNetworkGuru |
>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>09:11:44 |
>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>>-+
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>>+-
>>>>>>>>--
>>>>>>>>--
>>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>>--
>>>>>>>>--
>>>>>>>>--
>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>
>>>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
I don’t think its relevant as the piece we’ve fixed, just eliminated
static nat rule programming for non-existing vm. Missing netmask on eth1
doesn’t seem related to the problem (although we have to figure out why
its missing), as the connection that fails, happening to link local 169.x
eth0 interface.
Edison, can you please tell us how to debug link local connection failure,
on KVM agent?
Thank you,
Alena.
On 3/24/14, 1:47 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>thank you, the NullPointerException doesn't occur any more, but there
>still seems to be a problem during startup of the router.
>
>When I start the virtual router, it comes up, but in agent.log, there
>are lots of 'Could not connect to 169.254.1.x' messages.
>
>Then I logged into the virtual router to find out that the netmask of
>eth1 is missing in the interfaces file:
>
>root@host:~# virsh console r-19-VM
>Connected to domain r-19-VM
>Escape character is ^]
>
>Debian GNU/Linux 6.0 r-19-VM ttyS0
>
>r-19-VM login: root
>...
>root@r-19-VM:~# cat /etc/network/interfaces
>auto lo eth0 eth1 eth2
>iface lo inet loopback
>
>iface eth0 inet static
> address 169.254.1.59
> netmask 255.255.0.0
>iface eth1 inet static
> address 10.193.17.1
> netmask
>iface eth2 inet static
> address 10.193.17.190
> netmask 255.255.255.0
>
>I don't know if it is relevant, but this is the line from agent.log
>where the parameters are visible:
>
>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>(agentRequest-Handler-2:null) Executing:
>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga
>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou
>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.0%
>type=router%disable_rp_filter=true%dns1=10.193.17.1
>
>
>Any hint is appreciated.
>
>Thanks
>
>
>Kambiz
>
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
>> id=15 is expunged, we have to clear out the reference to it from
>> user_ip_address table. Here is the flow:
>>
>> 1) Save the db dump.
>> 2) Run the query to cleanup the reference:
>>
>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>> id=<problematic public ip address id>
>>
>>
>>
>> Let me know how it works.
>>
>> -Alena.
>>
>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>I hope I have understood what you wrote and created the following query
>>>correctly:
>>>
>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>> n.state as nic_state, n.removed as nic_removed,
>>> vm.state as vm_state, vm.removed as vm_removed
>>>from user_ip_address uip
>>> join nics n on uip.vm_id = n.instance_id
>>> join vm_instance vm on uip.vm_id = vm.id
>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>fr.network_id=205);
>>>
>>>
>>>+-------+------------+-------------------+--------------+---------------
>>>--
>>>----+-----------+------------+
>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>> | vm_state | vm_removed |
>>>+-------+------------+-------------------+--------------+---------------
>>>--
>>>----+-----------+------------+
>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>> | Stopped | NULL |
>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>> | Stopped | NULL |
>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>> | Stopped | NULL |
>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>> | Stopped | NULL |
>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>> | Stopped | NULL |
>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>23:00:53 | Expunging | NULL |
>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>> | Stopped | NULL |
>>>+-------+------------+-------------------+--------------+---------------
>>>--
>>>----+-----------+------------+
>>>
>>>Is VM id 15 what you are looking for?
>>>
>>>Thank you
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> Kambiz, can you please try one more thing.
>>>>
>>>> 1) Locate all the firewall rules for your guest network (205, right?)
>>>>
>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>
>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>
>>>> Select vm_id, network_id from user_ip_address where id in (Select id,
>>>> ip_address_id from firewall_rules where network_id=205);
>>>>
>>>> For each vmId/networkId combo, check if there is non-removed nic and
>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>reference
>>>> referring to vm that is removed already. If you find any, let me know
>>>>and
>>>> I will tell you how to clean it up
>>>>
>>>> -Alena.
>>>>
>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hi Alena,
>>>>>
>>>>>thank you for your help.
>>>>>
>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>removed
>>>>>the row though to see what happens: a new virtual router was created
>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>change by restoring from the dump.
>>>>>
>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>which
>>>>>was used by my instances. I deleted the router using the UI after the
>>>>>first
>>>>>occurrence of the NPE, because a post with a similar problem suggested
>>>>>that the deleted router would be recreated again (and this procedure
>>>>>solved the problem).
>>>>>
>>>>>Below I have attached the state of the two tables.
>>>>>
>>>>>Anything else I can try?
>>>>>
>>>>>Thank you
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>>>>>where i.type = 'DomainRouter';
>>>>>+----+---------------------+---------------+---------------+----------
>>>>>--
>>>>>-+
>>>>>---------+--------------------------+------------+-------------+------
>>>>>--
>>>>>-+
>>>>>-----------+--------------+
>>>>>| id | removed | ip4_address | netmask | gateway
>>>>>| ip_type | reserver_name | network_id | instance_id | name
>>>>>| state | type |
>>>>>+----+---------------------+---------------+---------------+----------
>>>>>--
>>>>>-+
>>>>>---------+--------------------------+------------+-------------+------
>>>>>--
>>>>>-+
>>>>>-----------+--------------+
>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>r-4-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>r-4-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>10.193.17.1
>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>r-4-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>r-7-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>r-7-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>10.193.17.1
>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>r-7-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>r-18-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>r-18-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>10.193.17.1
>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>r-18-VM
>>>>>| Expunging | DomainRouter |
>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>r-19-VM
>>>>>| Stopped | DomainRouter |
>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>r-19-VM
>>>>>| Stopped | DomainRouter |
>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>10.193.17.1
>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>r-19-VM
>>>>>| Stopped | DomainRouter |
>>>>>+----+---------------------+---------------+---------------+----------
>>>>>--
>>>>>-+
>>>>>---------+--------------------------+------------+-------------+------
>>>>>--
>>>>>-+
>>>>>-----------+--------------+
>>>>>
>>>>>mysql> select * from router_network_ref;
>>>>>+----+-----------+------------+------------+
>>>>>| id | router_id | network_id | guest_type |
>>>>>+----+-----------+------------+------------+
>>>>>| 1 | 4 | 204 | Isolated |
>>>>>| 2 | 7 | 205 | Isolated |
>>>>>| 3 | 18 | 205 | Isolated |
>>>>>| 4 | 19 | 205 | Isolated |
>>>>>+----+-----------+------------+------------+
>>>>>
>>>>>
>>>>>
>>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>>
>>>>>> The error happens not because Ip is null, but because the nic in a
>>>>>>certain
>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>> plug/unplug for Guest networks process.
>>>>>>
>>>>>> Kambiz, please do the following to fix it:
>>>>>>
>>>>>> 1) Stop the MS
>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>> 3) Run the query:
>>>>>>
>>>>>> select * from router_network_ref where router_id=<id of your VR) and
>>>>>> network_id not in (select network_id from nics where instance_id=<ID
>>>>>>of
>>>>>> your VR> and removed is null);
>>>>>>
>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>cleaned
>>>>>> during the nic detach. Remove the entry returned from
>>>>>>router_network_ref
>>>>>> table.
>>>>>>
>>>>>> Let me know how it works.
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>>
>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hello,
>>>>>>>
>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>contributors for Cloudstack which I use since last fall without any
>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>
>>>>>>>After a restart of the management server (stopping and starting the
>>>>>>>java
>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>
>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>out
>>>>>>>that the reason is a row in the table nics which has NULL in ip (cf.
>>>>>>>row
>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>
>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>suggestions
>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>
>>>>>>>Thank you for your help
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>
>>>>>>>management-server.log:
>>>>>>>
>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>>>>>with
>>>>>>>any VPC
>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>service
>>>>>>>Connectivity on network net1
>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in
>>>>>>>the
>>>>>>>network id=205
>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>part
>>>>>>>of
>>>>>>>network implement
>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>part
>>>>>>>of
>>>>>>>network implement
>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>associated
>>>>>>>with any VPC
>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>service
>>>>>>>Connectivity on network null
>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>for
>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>adding
>>>>>>>disk to VM
>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>eth2mask=255.255.255.0
>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>dns1=10.193.17.1
>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply
>>>>>>>as
>>>>>>>a
>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>java.lang.NullPointerException
>>>>>>> at
>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.j
>>>>>>>av
>>>>>>>a:
>>>>>>>76
>>>>>>>3)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>Ne
>>>>>>>tw
>>>>>>>or
>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>> at
>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final
>>>>>>>iz
>>>>>>>eN
>>>>>>>et
>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>Co
>>>>>>>mm
>>>>>>>an
>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>> at
>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final
>>>>>>>iz
>>>>>>>eC
>>>>>>>om
>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize
>>>>>>>De
>>>>>>>pl
>>>>>>>oy
>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>> at
>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMa
>>>>>>>na
>>>>>>>ge
>>>>>>>rI
>>>>>>>mpl.java:763)
>>>>>>> at
>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm
>>>>>>>pl
>>>>>>>.j
>>>>>>>av
>>>>>>>a:471)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Vi
>>>>>>>rt
>>>>>>>ua
>>>>>>>lN
>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVir
>>>>>>>tu
>>>>>>>al
>>>>>>>Ro
>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>te
>>>>>>>rs
>>>>>>>(V
>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVi
>>>>>>>rt
>>>>>>>ua
>>>>>>>lR
>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>> at
>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRout
>>>>>>>er
>>>>>>>El
>>>>>>>em
>>>>>>>ent.java:175)
>>>>>>> at
>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndReso
>>>>>>>ur
>>>>>>>ce
>>>>>>>s(
>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>> at
>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManager
>>>>>>>Im
>>>>>>>pl
>>>>>>>.j
>>>>>>>ava:1434)
>>>>>>> at
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>to
>>>>>>>rD
>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>> at
>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl
>>>>>>>.j
>>>>>>>av
>>>>>>>a:
>>>>>>>2435)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>te
>>>>>>>r(
>>>>>>>Vi
>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>> at
>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou
>>>>>>>te
>>>>>>>r(
>>>>>>>Vi
>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>> at
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>to
>>>>>>>rD
>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>> at
>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execut
>>>>>>>e(
>>>>>>>St
>>>>>>>ar
>>>>>>>tRouterCmd.java:103)
>>>>>>>
>>>>>>>
>>>>>>>table nics:
>>>>>>>
>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>'ControlNetworkGuru';
>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>-+
>>>>>>>--
>>>>>>>--
>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>+-
>>>>>>>--
>>>>>>>--
>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>| id | uuid | instance_id |
>>>>>>>mac_address
>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>broadcast_uri
>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>> |
>>>>>>>reservation_id | device_id | update_time
>>>>>>> |
>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>created
>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>-+
>>>>>>>--
>>>>>>>--
>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>+-
>>>>>>>--
>>>>>>>--
>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>>|
>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>0
>>>>>>>|
>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>NULL
>>>>>>> | NULL |
>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>>|
>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>0
>>>>>>>|
>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>NULL
>>>>>>> | NULL |
>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>ControlNetworkGuru |
>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>12:27:58 |
>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>ControlNetworkGuru |
>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>12:27:52 |
>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>ControlNetworkGuru |
>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>09:11:16 |
>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>ControlNetworkGuru |
>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>09:11:44 |
>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>>>>>+----+--------------------------------------+-------------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>----+---------------+-------------+-------------+---------+---------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>+------------+--------+--------------+----------+-------------------
>>>>>>>-+
>>>>>>>--
>>>>>>>--
>>>>>>>----------------------------------+-----------+---------------------
>>>>>>>+-
>>>>>>>--
>>>>>>>--
>>>>>>>----------+-------------+-------------+--------------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>--
>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>
>>>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
thank you, the NullPointerException doesn't occur any more, but there
still seems to be a problem during startup of the router.
When I start the virtual router, it comes up, but in agent.log, there
are lots of 'Could not connect to 169.254.1.x' messages.
Then I logged into the virtual router to find out that the netmask of
eth1 is missing in the interfaces file:
root@host:~# virsh console r-19-VM
Connected to domain r-19-VM
Escape character is ^]
Debian GNU/Linux 6.0 r-19-VM ttyS0
r-19-VM login: root
...
root@r-19-VM:~# cat /etc/network/interfaces
auto lo eth0 eth1 eth2
iface lo inet loopback
iface eth0 inet static
address 169.254.1.59
netmask 255.255.0.0
iface eth1 inet static
address 10.193.17.1
netmask
iface eth2 inet static
address 10.193.17.190
netmask 255.255.255.0
I don't know if it is relevant, but this is the line from agent.log
where the parameters are visible:
2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p %template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gateway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=10.193.17.1
Any hint is appreciated.
Thanks
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
> id=15 is expunged, we have to clear out the reference to it from
> user_ip_address table. Here is the flow:
>
> 1) Save the db dump.
> 2) Run the query to cleanup the reference:
>
> Update user_ip_address set one_to_one_nat=0, instance_id=null where
> id=<problematic public ip address id>
>
>
>
> Let me know how it works.
>
> -Alena.
>
> On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi,
>>
>>I hope I have understood what you wrote and created the following query
>>correctly:
>>
>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>> n.state as nic_state, n.removed as nic_removed,
>> vm.state as vm_state, vm.removed as vm_removed
>>from user_ip_address uip
>> join nics n on uip.vm_id = n.instance_id
>> join vm_instance vm on uip.vm_id = vm.id
>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>fr.network_id=205);
>>
>>
>>+-------+------------+-------------------+--------------+-----------------
>>----+-----------+------------+
>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>> | vm_state | vm_removed |
>>+-------+------------+-------------------+--------------+-----------------
>>----+-----------+------------+
>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>> | Stopped | NULL |
>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>> | Stopped | NULL |
>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>> | Stopped | NULL |
>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>> | Stopped | NULL |
>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>> | Stopped | NULL |
>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>23:00:53 | Expunging | NULL |
>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>> | Stopped | NULL |
>>+-------+------------+-------------------+--------------+-----------------
>>----+-----------+------------+
>>
>>Is VM id 15 what you are looking for?
>>
>>Thank you
>>
>>
>>Kambiz
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> Kambiz, can you please try one more thing.
>>>
>>> 1) Locate all the firewall rules for your guest network (205, right?)
>>>
>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>
>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>
>>> Select vm_id, network_id from user_ip_address where id in (Select id,
>>> ip_address_id from firewall_rules where network_id=205);
>>>
>>> For each vmId/networkId combo, check if there is non-removed nic and
>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>reference
>>> referring to vm that is removed already. If you find any, let me know
>>>and
>>> I will tell you how to clean it up
>>>
>>> -Alena.
>>>
>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hi Alena,
>>>>
>>>>thank you for your help.
>>>>
>>>>The query returns no rows, i.e. nics.removed was not null, but I removed
>>>>the row though to see what happens: a new virtual router was created
>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>change by restoring from the dump.
>>>>
>>>>I have to mention that prior to the restart, r-7-VM was the router which
>>>>was used by my instances. I deleted the router using the UI after the
>>>>first
>>>>occurrence of the NPE, because a post with a similar problem suggested
>>>>that the deleted router would be recreated again (and this procedure
>>>>solved the problem).
>>>>
>>>>Below I have attached the state of the two tables.
>>>>
>>>>Anything else I can try?
>>>>
>>>>Thank you
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>>>>where i.type = 'DomainRouter';
>>>>+----+---------------------+---------------+---------------+------------
>>>>-+
>>>>---------+--------------------------+------------+-------------+--------
>>>>-+
>>>>-----------+--------------+
>>>>| id | removed | ip4_address | netmask | gateway
>>>>| ip_type | reserver_name | network_id | instance_id | name
>>>>| state | type |
>>>>+----+---------------------+---------------+---------------+------------
>>>>-+
>>>>---------+--------------------------+------------+-------------+--------
>>>>-+
>>>>-----------+--------------+
>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM
>>>>| Expunging | DomainRouter |
>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>| NULL | ControlNetworkGuru | 202 | 4 | r-4-VM
>>>>| Expunging | DomainRouter |
>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1
>>>>| NULL | PublicNetworkGuru | 200 | 4 | r-4-VM
>>>>| Expunging | DomainRouter |
>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM
>>>>| Expunging | DomainRouter |
>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>| NULL | ControlNetworkGuru | 202 | 7 | r-7-VM
>>>>| Expunging | DomainRouter |
>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>>| NULL | PublicNetworkGuru | 200 | 7 | r-7-VM
>>>>| Expunging | DomainRouter |
>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>r-18-VM
>>>>| Expunging | DomainRouter |
>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>r-18-VM
>>>>| Expunging | DomainRouter |
>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>r-18-VM
>>>>| Expunging | DomainRouter |
>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>r-19-VM
>>>>| Stopped | DomainRouter |
>>>>| 30 | NULL | NULL | NULL | NULL
>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>r-19-VM
>>>>| Stopped | DomainRouter |
>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>r-19-VM
>>>>| Stopped | DomainRouter |
>>>>+----+---------------------+---------------+---------------+------------
>>>>-+
>>>>---------+--------------------------+------------+-------------+--------
>>>>-+
>>>>-----------+--------------+
>>>>
>>>>mysql> select * from router_network_ref;
>>>>+----+-----------+------------+------------+
>>>>| id | router_id | network_id | guest_type |
>>>>+----+-----------+------------+------------+
>>>>| 1 | 4 | 204 | Isolated |
>>>>| 2 | 7 | 205 | Isolated |
>>>>| 3 | 18 | 205 | Isolated |
>>>>| 4 | 19 | 205 | Isolated |
>>>>+----+-----------+------------+------------+
>>>>
>>>>
>>>>
>>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>>
>>>>> The error happens not because Ip is null, but because the nic in a
>>>>>certain
>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>> plug/unplug for Guest networks process.
>>>>>
>>>>> Kambiz, please do the following to fix it:
>>>>>
>>>>> 1) Stop the MS
>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>> 3) Run the query:
>>>>>
>>>>> select * from router_network_ref where router_id=<id of your VR) and
>>>>> network_id not in (select network_id from nics where instance_id=<ID
>>>>>of
>>>>> your VR> and removed is null);
>>>>>
>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>cleaned
>>>>> during the nic detach. Remove the entry returned from
>>>>>router_network_ref
>>>>> table.
>>>>>
>>>>> Let me know how it works.
>>>>>
>>>>> -Alena.
>>>>>
>>>>>
>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>>
>>>>>>Hello,
>>>>>>
>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>contributors for Cloudstack which I use since last fall without any
>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>
>>>>>>After a restart of the management server (stopping and starting the
>>>>>>java
>>>>>>process), the virtual domain router doesn't start and
>>>>>>management-server.log shows a NullPointerException in
>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>
>>>>>>By putting the server in debug mode and remote debugging, I found out
>>>>>>that the reason is a row in the table nics which has NULL in ip (cf.
>>>>>>row
>>>>>>with id 30 in the result of the select statement below).
>>>>>>
>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>suggestions
>>>>>>are appreciated as the system is currently unusable.
>>>>>>
>>>>>>Thank you for your help
>>>>>>
>>>>>>
>>>>>>Kambiz
>>>>>>
>>>>>>
>>>>>>management-server.log:
>>>>>>
>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>>>>with
>>>>>>any VPC
>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>service
>>>>>>Connectivity on network net1
>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>>>>>>network id=205
>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part
>>>>>>of
>>>>>>network implement
>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part
>>>>>>of
>>>>>>network implement
>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>>>>>>with any VPC
>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>service
>>>>>>Connectivity on network null
>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>>>>>>disk to VM
>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>dns1=10.193.17.1
>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>balancing rules as a part of Virtual router start
>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as
>>>>>>a
>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>java.lang.NullPointerException
>>>>>> at
>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.jav
>>>>>>a:
>>>>>>76
>>>>>>3)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNe
>>>>>>tw
>>>>>>or
>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>> at
>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz
>>>>>>eN
>>>>>>et
>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCo
>>>>>>mm
>>>>>>an
>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>> at
>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz
>>>>>>eC
>>>>>>om
>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDe
>>>>>>pl
>>>>>>oy
>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>> at
>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMana
>>>>>>ge
>>>>>>rI
>>>>>>mpl.java:763)
>>>>>> at
>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl
>>>>>>.j
>>>>>>av
>>>>>>a:471)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Virt
>>>>>>ua
>>>>>>lN
>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtu
>>>>>>al
>>>>>>Ro
>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>>rs
>>>>>>(V
>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirt
>>>>>>ua
>>>>>>lR
>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>> at
>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouter
>>>>>>El
>>>>>>em
>>>>>>ent.java:175)
>>>>>> at
>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResour
>>>>>>ce
>>>>>>s(
>>>>>>NetworkManagerImpl.java:1518)
>>>>>> at
>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerIm
>>>>>>pl
>>>>>>.j
>>>>>>ava:1434)
>>>>>> at
>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep
>>>>>>to
>>>>>>rD
>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>> at
>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.j
>>>>>>av
>>>>>>a:
>>>>>>2435)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>>r(
>>>>>>Vi
>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>> at
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>>r(
>>>>>>Vi
>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>> at
>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep
>>>>>>to
>>>>>>rD
>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>> at
>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(
>>>>>>St
>>>>>>ar
>>>>>>tRouterCmd.java:103)
>>>>>>
>>>>>>
>>>>>>table nics:
>>>>>>
>>>>>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>>>>>>+----+--------------------------------------+-------------+-----------
>>>>>>--
>>>>>>--
>>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>>--
>>>>>>--
>>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>>--
>>>>>>--
>>>>>>----------------------------------+-----------+---------------------+-
>>>>>>--
>>>>>>--
>>>>>>----------+-------------+-------------+--------------------+----------
>>>>>>--
>>>>>>--
>>>>>>-------+---------------------+-------------+----------+
>>>>>>| id | uuid | instance_id |
>>>>>>mac_address
>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>broadcast_uri
>>>>>>| network_id | mode | state | strategy | reserver_name |
>>>>>>reservation_id | device_id | update_time
>>>>>> |
>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>created
>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>+----+--------------------------------------+-------------+-----------
>>>>>>--
>>>>>>--
>>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>>--
>>>>>>--
>>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>>--
>>>>>>--
>>>>>>----------------------------------+-----------+---------------------+-
>>>>>>--
>>>>>>--
>>>>>>----------+-------------+-------------+--------------------+----------
>>>>>>--
>>>>>>--
>>>>>>-------+---------------------+-------------+----------+
>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>|
>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0
>>>>>>|
>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
>>>>>> | NULL |
>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>>|
>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0
>>>>>>|
>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
>>>>>> | NULL |
>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>12:27:58 |
>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>12:27:52 |
>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>09:11:16 |
>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>09:11:44 |
>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>>>>+----+--------------------------------------+-------------+-----------
>>>>>>--
>>>>>>--
>>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>>--
>>>>>>--
>>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>>--
>>>>>>--
>>>>>>----------------------------------+-----------+---------------------+-
>>>>>>--
>>>>>>--
>>>>>>----------+-------------+-------------+--------------------+----------
>>>>>>--
>>>>>>--
>>>>>>-------+---------------------+-------------+----------+
>>>>>>
>>>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
id=15 is expunged, we have to clear out the reference to it from
user_ip_address table. Here is the flow:
1) Save the db dump.
2) Run the query to cleanup the reference:
Update user_ip_address set one_to_one_nat=0, instance_id=null where
id=<problematic public ip address id>
Let me know how it works.
-Alena.
On 3/24/14, 10:55 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi,
>
>I hope I have understood what you wrote and created the following query
>correctly:
>
>select uip.vm_id, uip.network_id, uip.public_ip_address,
> n.state as nic_state, n.removed as nic_removed,
> vm.state as vm_state, vm.removed as vm_removed
>from user_ip_address uip
> join nics n on uip.vm_id = n.instance_id
> join vm_instance vm on uip.vm_id = vm.id
>where uip.id in (Select ip_address_id from firewall_rules fr where
>fr.network_id=205);
>
>
>+-------+------------+-------------------+--------------+-----------------
>----+-----------+------------+
>| vm_id | network_id | public_ip_address | nic_state | nic_removed
> | vm_state | vm_removed |
>+-------+------------+-------------------+--------------+-----------------
>----+-----------+------------+
>| 6 | 205 | 10.193.17.169 | Allocated | NULL
> | Stopped | NULL |
>| 10 | 205 | 10.193.17.136 | Allocated | NULL
> | Stopped | NULL |
>| 12 | 205 | 10.193.17.140 | Allocated | NULL
> | Stopped | NULL |
>| 13 | 205 | 10.193.17.141 | Allocated | NULL
> | Stopped | NULL |
>| 14 | 205 | 10.193.17.142 | Allocated | NULL
> | Stopped | NULL |
>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>23:00:53 | Expunging | NULL |
>| 16 | 205 | 10.193.17.103 | Allocated | NULL
> | Stopped | NULL |
>+-------+------------+-------------------+--------------+-----------------
>----+-----------+------------+
>
>Is VM id 15 what you are looking for?
>
>Thank you
>
>
>Kambiz
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> Kambiz, can you please try one more thing.
>>
>> 1) Locate all the firewall rules for your guest network (205, right?)
>>
>> Select id, ip_address_id from firewall_rules where network_id=205;
>>
>> 2) Now get all static nat enabled ip addresses for those rules:
>>
>> Select vm_id, network_id from user_ip_address where id in (Select id,
>> ip_address_id from firewall_rules where network_id=205);
>>
>> For each vmId/networkId combo, check if there is non-removed nic and
>> non-expunged vm. There might be some incorrect static nat ip/vm
>>reference
>> referring to vm that is removed already. If you find any, let me know
>>and
>> I will tell you how to clean it up
>>
>> -Alena.
>>
>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hi Alena,
>>>
>>>thank you for your help.
>>>
>>>The query returns no rows, i.e. nics.removed was not null, but I removed
>>>the row though to see what happens: a new virtual router was created
>>>which also couldn't be started due to the same NPE. I reverted the
>>>change by restoring from the dump.
>>>
>>>I have to mention that prior to the restart, r-7-VM was the router which
>>>was used by my instances. I deleted the router using the UI after the
>>>first
>>>occurrence of the NPE, because a post with a similar problem suggested
>>>that the deleted router would be recreated again (and this procedure
>>>solved the problem).
>>>
>>>Below I have attached the state of the two tables.
>>>
>>>Anything else I can try?
>>>
>>>Thank you
>>>
>>>
>>>Kambiz
>>>
>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>>>where i.type = 'DomainRouter';
>>>+----+---------------------+---------------+---------------+------------
>>>-+
>>>---------+--------------------------+------------+-------------+--------
>>>-+
>>>-----------+--------------+
>>>| id | removed | ip4_address | netmask | gateway
>>>| ip_type | reserver_name | network_id | instance_id | name
>>>| state | type |
>>>+----+---------------------+---------------+---------------+------------
>>>-+
>>>---------+--------------------------+------------+-------------+--------
>>>-+
>>>-----------+--------------+
>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM
>>>| Expunging | DomainRouter |
>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>| NULL | ControlNetworkGuru | 202 | 4 | r-4-VM
>>>| Expunging | DomainRouter |
>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1
>>>| NULL | PublicNetworkGuru | 200 | 4 | r-4-VM
>>>| Expunging | DomainRouter |
>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM
>>>| Expunging | DomainRouter |
>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>| NULL | ControlNetworkGuru | 202 | 7 | r-7-VM
>>>| Expunging | DomainRouter |
>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>| NULL | PublicNetworkGuru | 200 | 7 | r-7-VM
>>>| Expunging | DomainRouter |
>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>r-18-VM
>>>| Expunging | DomainRouter |
>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>r-18-VM
>>>| Expunging | DomainRouter |
>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>r-18-VM
>>>| Expunging | DomainRouter |
>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>r-19-VM
>>>| Stopped | DomainRouter |
>>>| 30 | NULL | NULL | NULL | NULL
>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>r-19-VM
>>>| Stopped | DomainRouter |
>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>r-19-VM
>>>| Stopped | DomainRouter |
>>>+----+---------------------+---------------+---------------+------------
>>>-+
>>>---------+--------------------------+------------+-------------+--------
>>>-+
>>>-----------+--------------+
>>>
>>>mysql> select * from router_network_ref;
>>>+----+-----------+------------+------------+
>>>| id | router_id | network_id | guest_type |
>>>+----+-----------+------------+------------+
>>>| 1 | 4 | 204 | Isolated |
>>>| 2 | 7 | 205 | Isolated |
>>>| 3 | 18 | 205 | Isolated |
>>>| 4 | 19 | 205 | Isolated |
>>>+----+-----------+------------+------------+
>>>
>>>
>>>
>>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>>
>>>> The error happens not because Ip is null, but because the nic in a
>>>>certain
>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>> plug/unplug for Guest networks process.
>>>>
>>>> Kambiz, please do the following to fix it:
>>>>
>>>> 1) Stop the MS
>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>> 3) Run the query:
>>>>
>>>> select * from router_network_ref where router_id=<id of your VR) and
>>>> network_id not in (select network_id from nics where instance_id=<ID
>>>>of
>>>> your VR> and removed is null);
>>>>
>>>> It will give you the list of networks refs that somehow weren¹t
>>>>cleaned
>>>> during the nic detach. Remove the entry returned from
>>>>router_network_ref
>>>> table.
>>>>
>>>> Let me know how it works.
>>>>
>>>> -Alena.
>>>>
>>>>
>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>>
>>>>>Hello,
>>>>>
>>>>>as this is my first post to the list, I would like to thank all
>>>>>contributors for Cloudstack which I use since last fall without any
>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>
>>>>>After a restart of the management server (stopping and starting the
>>>>>java
>>>>>process), the virtual domain router doesn't start and
>>>>>management-server.log shows a NullPointerException in
>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>
>>>>>By putting the server in debug mode and remote debugging, I found out
>>>>>that the reason is a row in the table nics which has NULL in ip (cf.
>>>>>row
>>>>>with id 30 in the result of the select statement below).
>>>>>
>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>suggestions
>>>>>are appreciated as the system is currently unusable.
>>>>>
>>>>>Thank you for your help
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>
>>>>>management-server.log:
>>>>>
>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>2014-03-18 10:03:27,151 WARN
>>>>>[network.element.VpcVirtualRouterElement]
>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>>>with
>>>>>any VPC
>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>service
>>>>>Connectivity on network net1
>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>>>>>network id=205
>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part
>>>>>of
>>>>>network implement
>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part
>>>>>of
>>>>>network implement
>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>2014-03-18 10:03:27,187 WARN
>>>>>[network.element.VpcVirtualRouterElement]
>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>>>>>with any VPC
>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>service
>>>>>Connectivity on network null
>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>>>>>VM[DomainRouter|r-19-VM]
>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>>>>>disk to VM
>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>dns1=10.193.17.1
>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>balancing rules as a part of Virtual router start
>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as
>>>>>a
>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>VM[DomainRouter|r-19-VM]
>>>>>java.lang.NullPointerException
>>>>> at
>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.jav
>>>>>a:
>>>>>76
>>>>>3)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNe
>>>>>tw
>>>>>or
>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>> at
>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz
>>>>>eN
>>>>>et
>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCo
>>>>>mm
>>>>>an
>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>> at
>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz
>>>>>eC
>>>>>om
>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDe
>>>>>pl
>>>>>oy
>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>> at
>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMana
>>>>>ge
>>>>>rI
>>>>>mpl.java:763)
>>>>> at
>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl
>>>>>.j
>>>>>av
>>>>>a:471)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Virt
>>>>>ua
>>>>>lN
>>>>>etworkApplianceManagerImpl.java:2616)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtu
>>>>>al
>>>>>Ro
>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>rs
>>>>>(V
>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirt
>>>>>ua
>>>>>lR
>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>> at
>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouter
>>>>>El
>>>>>em
>>>>>ent.java:175)
>>>>> at
>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResour
>>>>>ce
>>>>>s(
>>>>>NetworkManagerImpl.java:1518)
>>>>> at
>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerIm
>>>>>pl
>>>>>.j
>>>>>ava:1434)
>>>>> at
>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep
>>>>>to
>>>>>rD
>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>> at
>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.j
>>>>>av
>>>>>a:
>>>>>2435)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>r(
>>>>>Vi
>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>> at
>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute
>>>>>r(
>>>>>Vi
>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>> at
>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep
>>>>>to
>>>>>rD
>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>> at
>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(
>>>>>St
>>>>>ar
>>>>>tRouterCmd.java:103)
>>>>>
>>>>>
>>>>>table nics:
>>>>>
>>>>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>>>>>+----+--------------------------------------+-------------+-----------
>>>>>--
>>>>>--
>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>--
>>>>>--
>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>--
>>>>>--
>>>>>----------------------------------+-----------+---------------------+-
>>>>>--
>>>>>--
>>>>>----------+-------------+-------------+--------------------+----------
>>>>>--
>>>>>--
>>>>>-------+---------------------+-------------+----------+
>>>>>| id | uuid | instance_id |
>>>>>mac_address
>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>broadcast_uri
>>>>>| network_id | mode | state | strategy | reserver_name |
>>>>>reservation_id | device_id | update_time
>>>>> |
>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>created
>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>+----+--------------------------------------+-------------+-----------
>>>>>--
>>>>>--
>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>--
>>>>>--
>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>--
>>>>>--
>>>>>----------------------------------+-----------+---------------------+-
>>>>>--
>>>>>--
>>>>>----------+-------------+-------------+--------------------+----------
>>>>>--
>>>>>--
>>>>>-------+---------------------+-------------+----------+
>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>|
>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0
>>>>>|
>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
>>>>> | NULL |
>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>>|
>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0
>>>>>|
>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
>>>>> | NULL |
>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>12:27:58 |
>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>12:27:52 |
>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>09:11:16 |
>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>09:11:44 |
>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>>>+----+--------------------------------------+-------------+-----------
>>>>>--
>>>>>--
>>>>>----+---------------+-------------+-------------+---------+-----------
>>>>>--
>>>>>--
>>>>>+------------+--------+--------------+----------+--------------------+
>>>>>--
>>>>>--
>>>>>----------------------------------+-----------+---------------------+-
>>>>>--
>>>>>--
>>>>>----------+-------------+-------------+--------------------+----------
>>>>>--
>>>>>--
>>>>>-------+---------------------+-------------+----------+
>>>>>
>>>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi,
I hope I have understood what you wrote and created the following query
correctly:
select uip.vm_id, uip.network_id, uip.public_ip_address,
n.state as nic_state, n.removed as nic_removed,
vm.state as vm_state, vm.removed as vm_removed
from user_ip_address uip
join nics n on uip.vm_id = n.instance_id
join vm_instance vm on uip.vm_id = vm.id
where uip.id in (Select ip_address_id from firewall_rules fr where fr.network_id=205);
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
| vm_id | network_id | public_ip_address | nic_state | nic_removed | vm_state | vm_removed |
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
| 6 | 205 | 10.193.17.169 | Allocated | NULL | Stopped | NULL |
| 10 | 205 | 10.193.17.136 | Allocated | NULL | Stopped | NULL |
| 12 | 205 | 10.193.17.140 | Allocated | NULL | Stopped | NULL |
| 13 | 205 | 10.193.17.141 | Allocated | NULL | Stopped | NULL |
| 14 | 205 | 10.193.17.142 | Allocated | NULL | Stopped | NULL |
| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18 23:00:53 | Expunging | NULL |
| 16 | 205 | 10.193.17.103 | Allocated | NULL | Stopped | NULL |
+-------+------------+-------------------+--------------+---------------------+-----------+------------+
Is VM id 15 what you are looking for?
Thank you
Kambiz
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> Kambiz, can you please try one more thing.
>
> 1) Locate all the firewall rules for your guest network (205, right?)
>
> Select id, ip_address_id from firewall_rules where network_id=205;
>
> 2) Now get all static nat enabled ip addresses for those rules:
>
> Select vm_id, network_id from user_ip_address where id in (Select id,
> ip_address_id from firewall_rules where network_id=205);
>
> For each vmId/networkId combo, check if there is non-removed nic and
> non-expunged vm. There might be some incorrect static nat ip/vm reference
> referring to vm that is removed already. If you find any, let me know and
> I will tell you how to clean it up
>
> -Alena.
>
> On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hi Alena,
>>
>>thank you for your help.
>>
>>The query returns no rows, i.e. nics.removed was not null, but I removed
>>the row though to see what happens: a new virtual router was created
>>which also couldn't be started due to the same NPE. I reverted the
>>change by restoring from the dump.
>>
>>I have to mention that prior to the restart, r-7-VM was the router which
>>was used by my instances. I deleted the router using the UI after the
>>first
>>occurrence of the NPE, because a post with a similar problem suggested
>>that the deleted router would be recreated again (and this procedure
>>solved the problem).
>>
>>Below I have attached the state of the two tables.
>>
>>Anything else I can try?
>>
>>Thank you
>>
>>
>>Kambiz
>>
>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>>where i.type = 'DomainRouter';
>>+----+---------------------+---------------+---------------+-------------+
>>---------+--------------------------+------------+-------------+---------+
>>-----------+--------------+
>>| id | removed | ip4_address | netmask | gateway
>>| ip_type | reserver_name | network_id | instance_id | name
>>| state | type |
>>+----+---------------------+---------------+---------------+-------------+
>>---------+--------------------------+------------+-------------+---------+
>>-----------+--------------+
>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM
>>| Expunging | DomainRouter |
>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>| NULL | ControlNetworkGuru | 202 | 4 | r-4-VM
>>| Expunging | DomainRouter |
>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1
>>| NULL | PublicNetworkGuru | 200 | 4 | r-4-VM
>>| Expunging | DomainRouter |
>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM
>>| Expunging | DomainRouter |
>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>| NULL | ControlNetworkGuru | 202 | 7 | r-7-VM
>>| Expunging | DomainRouter |
>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>| NULL | PublicNetworkGuru | 200 | 7 | r-7-VM
>>| Expunging | DomainRouter |
>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | r-18-VM
>>| Expunging | DomainRouter |
>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>| NULL | ControlNetworkGuru | 202 | 18 | r-18-VM
>>| Expunging | DomainRouter |
>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>| NULL | PublicNetworkGuru | 200 | 18 | r-18-VM
>>| Expunging | DomainRouter |
>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | r-19-VM
>>| Stopped | DomainRouter |
>>| 30 | NULL | NULL | NULL | NULL
>>| NULL | ControlNetworkGuru | 202 | 19 | r-19-VM
>>| Stopped | DomainRouter |
>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>>| NULL | PublicNetworkGuru | 200 | 19 | r-19-VM
>>| Stopped | DomainRouter |
>>+----+---------------------+---------------+---------------+-------------+
>>---------+--------------------------+------------+-------------+---------+
>>-----------+--------------+
>>
>>mysql> select * from router_network_ref;
>>+----+-----------+------------+------------+
>>| id | router_id | network_id | guest_type |
>>+----+-----------+------------+------------+
>>| 1 | 4 | 204 | Isolated |
>>| 2 | 7 | 205 | Isolated |
>>| 3 | 18 | 205 | Isolated |
>>| 4 | 19 | 205 | Isolated |
>>+----+-----------+------------+------------+
>>
>>
>>
>>Alena Prokharchyk <Al...@citrix.com> wrote:
>>>
>>> The error happens not because Ip is null, but because the nic in a
>>>certain
>>> network can¹t be found. Looks like there is some bug in VPC nic
>>> plug/unplug for Guest networks process.
>>>
>>> Kambiz, please do the following to fix it:
>>>
>>> 1) Stop the MS
>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>> 3) Run the query:
>>>
>>> select * from router_network_ref where router_id=<id of your VR) and
>>> network_id not in (select network_id from nics where instance_id=<ID of
>>> your VR> and removed is null);
>>>
>>> It will give you the list of networks refs that somehow weren¹t cleaned
>>> during the nic detach. Remove the entry returned from router_network_ref
>>> table.
>>>
>>> Let me know how it works.
>>>
>>> -Alena.
>>>
>>>
>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>>
>>>>Hello,
>>>>
>>>>as this is my first post to the list, I would like to thank all
>>>>contributors for Cloudstack which I use since last fall without any
>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>
>>>>After a restart of the management server (stopping and starting the java
>>>>process), the virtual domain router doesn't start and
>>>>management-server.log shows a NullPointerException in
>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>
>>>>By putting the server in debug mode and remote debugging, I found out
>>>>that the reason is a row in the table nics which has NULL in ip (cf. row
>>>>with id 30 in the result of the select statement below).
>>>>
>>>>What can I do to quickly solve this problem? Any pointers or suggestions
>>>>are appreciated as the system is currently unusable.
>>>>
>>>>Thank you for your help
>>>>
>>>>
>>>>Kambiz
>>>>
>>>>
>>>>management-server.log:
>>>>
>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement]
>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>>with
>>>>any VPC
>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>>>Connectivity on network net1
>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>>>>network id=205
>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part
>>>>of
>>>>network implement
>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part
>>>>of
>>>>network implement
>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement]
>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>>>>with any VPC
>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>>>Connectivity on network null
>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>>>>VM[DomainRouter|r-19-VM]
>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>>>>disk to VM
>>>>2014-03-18 10:03:27,224 DEBUG
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1
>>>>2014-03-18 10:03:27,343 DEBUG
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>VM[DomainRouter|r-19-VM] start.
>>>>2014-03-18 10:03:27,415 DEBUG
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>balancing rules as a part of Virtual router start
>>>>2014-03-18 10:03:27,499 DEBUG
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a
>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>VM[DomainRouter|r-19-VM]
>>>>java.lang.NullPointerException
>>>> at
>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:
>>>>76
>>>>3)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetw
>>>>or
>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>> at
>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeN
>>>>et
>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeComm
>>>>an
>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>> at
>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeC
>>>>om
>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDepl
>>>>oy
>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>> at
>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManage
>>>>rI
>>>>mpl.java:763)
>>>> at
>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.j
>>>>av
>>>>a:471)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Virtua
>>>>lN
>>>>etworkApplianceManagerImpl.java:2616)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtual
>>>>Ro
>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters
>>>>(V
>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtua
>>>>lR
>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>> at
>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterEl
>>>>em
>>>>ent.java:175)
>>>> at
>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResource
>>>>s(
>>>>NetworkManagerImpl.java:1518)
>>>> at
>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl
>>>>.j
>>>>ava:1434)
>>>> at
>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercepto
>>>>rD
>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>> at
>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.jav
>>>>a:
>>>>2435)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(
>>>>Vi
>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>> at
>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(
>>>>Vi
>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>> at
>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercepto
>>>>rD
>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>> at
>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(St
>>>>ar
>>>>tRouterCmd.java:103)
>>>>
>>>>
>>>>table nics:
>>>>
>>>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>>>>+----+--------------------------------------+-------------+-------------
>>>>--
>>>>----+---------------+-------------+-------------+---------+-------------
>>>>--
>>>>+------------+--------+--------------+----------+--------------------+--
>>>>--
>>>>----------------------------------+-----------+---------------------+---
>>>>--
>>>>----------+-------------+-------------+--------------------+------------
>>>>--
>>>>-------+---------------------+-------------+----------+
>>>>| id | uuid | instance_id | mac_address
>>>> | ip4_address | netmask | gateway | ip_type |
>>>>broadcast_uri
>>>>| network_id | mode | state | strategy | reserver_name |
>>>>reservation_id | device_id | update_time |
>>>>isolation_uri | ip6_address | default_nic | vm_type | created
>>>> | removed | ip6_gateway | ip6_cidr |
>>>>+----+--------------------------------------+-------------+-------------
>>>>--
>>>>----+---------------+-------------+-------------+---------+-------------
>>>>--
>>>>+------------+--------+--------------+----------+--------------------+--
>>>>--
>>>>----------------------------------+-----------+---------------------+---
>>>>--
>>>>----------+-------------+-------------+--------------------+------------
>>>>--
>>>>-------+---------------------+-------------+----------+
>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>|
>>>>NULL | 202 | Static | Reserved | Start |
>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 |
>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
>>>> | NULL |
>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>>|
>>>>NULL | 202 | Static | Reserved | Start |
>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 |
>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
>>>> | NULL |
>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>> | NULL | NULL | NULL | NULL | NULL
>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 |
>>>>NULL | NULL | 0 | DomainRouter |
>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>> | NULL | NULL | NULL | NULL | NULL
>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 |
>>>>NULL | NULL | 0 | DomainRouter |
>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>> | NULL | NULL | NULL | NULL | NULL
>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 |
>>>>NULL | NULL | 0 | DomainRouter |
>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>> | NULL | NULL | NULL | NULL | NULL
>>>>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 |
>>>>NULL | NULL | 0 | DomainRouter |
>>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>>+----+--------------------------------------+-------------+-------------
>>>>--
>>>>----+---------------+-------------+-------------+---------+-------------
>>>>--
>>>>+------------+--------+--------------+----------+--------------------+--
>>>>--
>>>>----------------------------------+-----------+---------------------+---
>>>>--
>>>>----------+-------------+-------------+--------------------+------------
>>>>--
>>>>-------+---------------------+-------------+----------+
>>>>
>>>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
Kambiz, can you please try one more thing.
1) Locate all the firewall rules for your guest network (205, right?)
Select id, ip_address_id from firewall_rules where network_id=205;
2) Now get all static nat enabled ip addresses for those rules:
Select vm_id, network_id from user_ip_address where id in (Select id,
ip_address_id from firewall_rules where network_id=205);
For each vmId/networkId combo, check if there is non-removed nic and
non-expunged vm. There might be some incorrect static nat ip/vm reference
referring to vm that is removed already. If you find any, let me know and
I will tell you how to clean it up
-Alena.
On 3/22/14, 5:41 AM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hi Alena,
>
>thank you for your help.
>
>The query returns no rows, i.e. nics.removed was not null, but I removed
>the row though to see what happens: a new virtual router was created
>which also couldn't be started due to the same NPE. I reverted the
>change by restoring from the dump.
>
>I have to mention that prior to the restart, r-7-VM was the router which
>was used by my instances. I deleted the router using the UI after the
>first
>occurrence of the NPE, because a post with a similar problem suggested
>that the deleted router would be recreated again (and this procedure
>solved the problem).
>
>Below I have attached the state of the two tables.
>
>Anything else I can try?
>
>Thank you
>
>
>Kambiz
>
>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name,
>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id
>where i.type = 'DomainRouter';
>+----+---------------------+---------------+---------------+-------------+
>---------+--------------------------+------------+-------------+---------+
>-----------+--------------+
>| id | removed | ip4_address | netmask | gateway
>| ip_type | reserver_name | network_id | instance_id | name
>| state | type |
>+----+---------------------+---------------+---------------+-------------+
>---------+--------------------------+------------+-------------+---------+
>-----------+--------------+
>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>| NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM
>| Expunging | DomainRouter |
>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>| NULL | ControlNetworkGuru | 202 | 4 | r-4-VM
>| Expunging | DomainRouter |
>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1
>| NULL | PublicNetworkGuru | 200 | 4 | r-4-VM
>| Expunging | DomainRouter |
>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>| NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM
>| Expunging | DomainRouter |
>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>| NULL | ControlNetworkGuru | 202 | 7 | r-7-VM
>| Expunging | DomainRouter |
>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>| NULL | PublicNetworkGuru | 200 | 7 | r-7-VM
>| Expunging | DomainRouter |
>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>| NULL | ExternalGuestNetworkGuru | 205 | 18 | r-18-VM
>| Expunging | DomainRouter |
>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>| NULL | ControlNetworkGuru | 202 | 18 | r-18-VM
>| Expunging | DomainRouter |
>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>| NULL | PublicNetworkGuru | 200 | 18 | r-18-VM
>| Expunging | DomainRouter |
>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>| NULL | ExternalGuestNetworkGuru | 205 | 19 | r-19-VM
>| Stopped | DomainRouter |
>| 30 | NULL | NULL | NULL | NULL
>| NULL | ControlNetworkGuru | 202 | 19 | r-19-VM
>| Stopped | DomainRouter |
>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1
>| NULL | PublicNetworkGuru | 200 | 19 | r-19-VM
>| Stopped | DomainRouter |
>+----+---------------------+---------------+---------------+-------------+
>---------+--------------------------+------------+-------------+---------+
>-----------+--------------+
>
>mysql> select * from router_network_ref;
>+----+-----------+------------+------------+
>| id | router_id | network_id | guest_type |
>+----+-----------+------------+------------+
>| 1 | 4 | 204 | Isolated |
>| 2 | 7 | 205 | Isolated |
>| 3 | 18 | 205 | Isolated |
>| 4 | 19 | 205 | Isolated |
>+----+-----------+------------+------------+
>
>
>
>Alena Prokharchyk <Al...@citrix.com> wrote:
>>
>> The error happens not because Ip is null, but because the nic in a
>>certain
>> network can¹t be found. Looks like there is some bug in VPC nic
>> plug/unplug for Guest networks process.
>>
>> Kambiz, please do the following to fix it:
>>
>> 1) Stop the MS
>> 2) Take the DB dump of cloud db in case you have to revert back.
>> 3) Run the query:
>>
>> select * from router_network_ref where router_id=<id of your VR) and
>> network_id not in (select network_id from nics where instance_id=<ID of
>> your VR> and removed is null);
>>
>> It will give you the list of networks refs that somehow weren¹t cleaned
>> during the nic detach. Remove the entry returned from router_network_ref
>> table.
>>
>> Let me know how it works.
>>
>> -Alena.
>>
>>
>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>>
>>>Hello,
>>>
>>>as this is my first post to the list, I would like to thank all
>>>contributors for Cloudstack which I use since last fall without any
>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>
>>>After a restart of the management server (stopping and starting the java
>>>process), the virtual domain router doesn't start and
>>>management-server.log shows a NullPointerException in
>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>
>>>By putting the server in debug mode and remote debugging, I found out
>>>that the reason is a row in the table nics which has NULL in ip (cf. row
>>>with id 30 in the result of the select statement below).
>>>
>>>What can I do to quickly solve this problem? Any pointers or suggestions
>>>are appreciated as the system is currently unusable.
>>>
>>>Thank you for your help
>>>
>>>
>>>Kambiz
>>>
>>>
>>>management-server.log:
>>>
>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement]
>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated
>>>with
>>>any VPC
>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>>Connectivity on network net1
>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>>>network id=205
>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part
>>>of
>>>network implement
>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part
>>>of
>>>network implement
>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement]
>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>>>with any VPC
>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>>Connectivity on network null
>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>>>VM[DomainRouter|r-19-VM]
>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>>>disk to VM
>>>2014-03-18 10:03:27,224 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1
>>>2014-03-18 10:03:27,343 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>VM[DomainRouter|r-19-VM] start.
>>>2014-03-18 10:03:27,415 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>balancing rules as a part of Virtual router start
>>>2014-03-18 10:03:27,499 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a
>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>(Job-Executor-1:job-176) Failed to start instance
>>>VM[DomainRouter|r-19-VM]
>>>java.lang.NullPointerException
>>> at
>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:
>>>76
>>>3)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetw
>>>or
>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>> at
>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeN
>>>et
>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeComm
>>>an
>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>> at
>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeC
>>>om
>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDepl
>>>oy
>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>> at
>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManage
>>>rI
>>>mpl.java:763)
>>> at
>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.j
>>>av
>>>a:471)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Virtua
>>>lN
>>>etworkApplianceManagerImpl.java:2616)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtual
>>>Ro
>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters
>>>(V
>>>irtualNetworkApplianceManagerImpl.java:1924)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtua
>>>lR
>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>> at
>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterEl
>>>em
>>>ent.java:175)
>>> at
>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResource
>>>s(
>>>NetworkManagerImpl.java:1518)
>>> at
>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl
>>>.j
>>>ava:1434)
>>> at
>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercepto
>>>rD
>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>> at
>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.jav
>>>a:
>>>2435)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(
>>>Vi
>>>rtualNetworkApplianceManagerImpl.java:2855)
>>> at
>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(
>>>Vi
>>>rtualNetworkApplianceManagerImpl.java:2824)
>>> at
>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercepto
>>>rD
>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>> at
>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(St
>>>ar
>>>tRouterCmd.java:103)
>>>
>>>
>>>table nics:
>>>
>>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>>>+----+--------------------------------------+-------------+-------------
>>>--
>>>----+---------------+-------------+-------------+---------+-------------
>>>--
>>>+------------+--------+--------------+----------+--------------------+--
>>>--
>>>----------------------------------+-----------+---------------------+---
>>>--
>>>----------+-------------+-------------+--------------------+------------
>>>--
>>>-------+---------------------+-------------+----------+
>>>| id | uuid | instance_id | mac_address
>>> | ip4_address | netmask | gateway | ip_type |
>>>broadcast_uri
>>>| network_id | mode | state | strategy | reserver_name |
>>>reservation_id | device_id | update_time |
>>>isolation_uri | ip6_address | default_nic | vm_type | created
>>> | removed | ip6_gateway | ip6_cidr |
>>>+----+--------------------------------------+-------------+-------------
>>>--
>>>----+---------------+-------------+-------------+---------+-------------
>>>--
>>>+------------+--------+--------------+----------+--------------------+--
>>>--
>>>----------------------------------+-----------+---------------------+---
>>>--
>>>----------+-------------+-------------+--------------------+------------
>>>--
>>>-------+---------------------+-------------+----------+
>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>|
>>>NULL | 202 | Static | Reserved | Start |
>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 |
>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
>>> | NULL |
>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4
>>>|
>>>NULL | 202 | Static | Reserved | Start |
>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 |
>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
>>> | NULL |
>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>> | NULL | NULL | NULL | NULL | NULL
>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 |
>>>NULL | NULL | 0 | DomainRouter |
>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>> | NULL | NULL | NULL | NULL | NULL
>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 |
>>>NULL | NULL | 0 | DomainRouter |
>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>> | NULL | NULL | NULL | NULL | NULL
>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 |
>>>NULL | NULL | 0 | DomainRouter |
>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>> | NULL | NULL | NULL | NULL | NULL
>>>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 |
>>>NULL | NULL | 0 | DomainRouter |
>>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>>+----+--------------------------------------+-------------+-------------
>>>--
>>>----+---------------+-------------+-------------+---------+-------------
>>>--
>>>+------------+--------+--------------+----------+--------------------+--
>>>--
>>>----------------------------------+-----------+---------------------+---
>>>--
>>>----------+-------------+-------------+--------------------+------------
>>>--
>>>-------+---------------------+-------------+----------+
>>>
>>>
Re: Virtual Router doesn't start
Posted by Kambiz Darabi <da...@m-creations.com>.
Hi Alena,
thank you for your help.
The query returns no rows, i.e. nics.removed was not null, but I removed
the row though to see what happens: a new virtual router was created
which also couldn't be started due to the same NPE. I reverted the
change by restoring from the dump.
I have to mention that prior to the restart, r-7-VM was the router which
was used by my instances. I deleted the router using the UI after the first
occurrence of the NPE, because a post with a similar problem suggested
that the deleted router would be recreated again (and this procedure
solved the problem).
Below I have attached the state of the two tables.
Anything else I can try?
Thank you
Kambiz
mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway, n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name, i.state, i.type from vm_instance i join nics n on n.instance_id = i.id where i.type = 'DomainRouter';
+----+---------------------+---------------+---------------+-------------+---------+--------------------------+------------+-------------+---------+-----------+--------------+
| id | removed | ip4_address | netmask | gateway | ip_type | reserver_name | network_id | instance_id | name | state | type |
+----+---------------------+---------------+---------------+-------------+---------+--------------------------+------------+-------------+---------+-----------+--------------+
| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL | NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM | Expunging | DomainRouter |
| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL | NULL | ControlNetworkGuru | 202 | 4 | r-4-VM | Expunging | DomainRouter |
| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1 | NULL | PublicNetworkGuru | 200 | 4 | r-4-VM | Expunging | DomainRouter |
| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL | NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM | Expunging | DomainRouter |
| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL | NULL | ControlNetworkGuru | 202 | 7 | r-7-VM | Expunging | DomainRouter |
| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | NULL | PublicNetworkGuru | 200 | 7 | r-7-VM | Expunging | DomainRouter |
| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL | NULL | ExternalGuestNetworkGuru | 205 | 18 | r-18-VM | Expunging | DomainRouter |
| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL | NULL | ControlNetworkGuru | 202 | 18 | r-18-VM | Expunging | DomainRouter |
| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | NULL | PublicNetworkGuru | 200 | 18 | r-18-VM | Expunging | DomainRouter |
| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL | NULL | ExternalGuestNetworkGuru | 205 | 19 | r-19-VM | Stopped | DomainRouter |
| 30 | NULL | NULL | NULL | NULL | NULL | ControlNetworkGuru | 202 | 19 | r-19-VM | Stopped | DomainRouter |
| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | NULL | PublicNetworkGuru | 200 | 19 | r-19-VM | Stopped | DomainRouter |
+----+---------------------+---------------+---------------+-------------+---------+--------------------------+------------+-------------+---------+-----------+--------------+
mysql> select * from router_network_ref;
+----+-----------+------------+------------+
| id | router_id | network_id | guest_type |
+----+-----------+------------+------------+
| 1 | 4 | 204 | Isolated |
| 2 | 7 | 205 | Isolated |
| 3 | 18 | 205 | Isolated |
| 4 | 19 | 205 | Isolated |
+----+-----------+------------+------------+
Alena Prokharchyk <Al...@citrix.com> wrote:
>
> The error happens not because Ip is null, but because the nic in a certain
> network can¹t be found. Looks like there is some bug in VPC nic
> plug/unplug for Guest networks process.
>
> Kambiz, please do the following to fix it:
>
> 1) Stop the MS
> 2) Take the DB dump of cloud db in case you have to revert back.
> 3) Run the query:
>
> select * from router_network_ref where router_id=<id of your VR) and
> network_id not in (select network_id from nics where instance_id=<ID of
> your VR> and removed is null);
>
> It will give you the list of networks refs that somehow weren¹t cleaned
> during the nic detach. Remove the entry returned from router_network_ref
> table.
>
> Let me know how it works.
>
> -Alena.
>
>
> On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>
>>Hello,
>>
>>as this is my first post to the list, I would like to thank all
>>contributors for Cloudstack which I use since last fall without any
>>problems. I run 4.1.1 with KVM and advanced networking.
>>
>>After a restart of the management server (stopping and starting the java
>>process), the virtual domain router doesn't start and
>>management-server.log shows a NullPointerException in
>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>
>>By putting the server in debug mode and remote debugging, I found out
>>that the reason is a row in the table nics which has NULL in ip (cf. row
>>with id 30 in the result of the select statement below).
>>
>>What can I do to quickly solve this problem? Any pointers or suggestions
>>are appreciated as the system is currently unusable.
>>
>>Thank you for your help
>>
>>
>>Kambiz
>>
>>
>>management-server.log:
>>
>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement]
>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated with
>>any VPC
>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>Connectivity on network net1
>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>>network id=205
>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part of
>>network implement
>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part of
>>network implement
>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement]
>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>>with any VPC
>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>>Connectivity on network null
>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>>VM[DomainRouter|r-19-VM]
>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>(Job-Executor-1:job-176) No need to recreate the volume:
>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>>disk to VM
>>2014-03-18 10:03:27,224 DEBUG
>>[network.router.VirtualNetworkApplianceManagerImpl]
>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1
>>2014-03-18 10:03:27,343 DEBUG
>>[network.router.VirtualNetworkApplianceManagerImpl]
>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>VM[DomainRouter|r-19-VM] start.
>>2014-03-18 10:03:27,415 DEBUG
>>[network.router.VirtualNetworkApplianceManagerImpl]
>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>balancing rules as a part of Virtual router start
>>2014-03-18 10:03:27,499 DEBUG
>>[network.router.VirtualNetworkApplianceManagerImpl]
>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a
>>part of domR VM[DomainRouter|r-19-VM] start.
>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>(Job-Executor-1:job-176) Failed to start instance VM[DomainRouter|r-19-VM]
>>java.lang.NullPointerException
>> at
>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:76
>>3)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetwor
>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>> at
>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNet
>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeComman
>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>> at
>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCom
>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeploy
>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>> at
>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerI
>>mpl.java:763)
>> at
>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav
>>a:471)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualN
>>etworkApplianceManagerImpl.java:2616)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRo
>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(V
>>irtualNetworkApplianceManagerImpl.java:1924)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualR
>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>> at
>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElem
>>ent.java:175)
>> at
>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(
>>NetworkManagerImpl.java:1518)
>> at
>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.j
>>ava:1434)
>> at
>>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> at
>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:
>>2435)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi
>>rtualNetworkApplianceManagerImpl.java:2855)
>> at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi
>>rtualNetworkApplianceManagerImpl.java:2824)
>> at
>>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> at
>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(Star
>>tRouterCmd.java:103)
>>
>>
>>table nics:
>>
>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>>+----+--------------------------------------+-------------+---------------
>>----+---------------+-------------+-------------+---------+---------------
>>+------------+--------+--------------+----------+--------------------+----
>>----------------------------------+-----------+---------------------+-----
>>----------+-------------+-------------+--------------------+--------------
>>-------+---------------------+-------------+----------+
>>| id | uuid | instance_id | mac_address
>> | ip4_address | netmask | gateway | ip_type | broadcast_uri
>>| network_id | mode | state | strategy | reserver_name |
>>reservation_id | device_id | update_time |
>>isolation_uri | ip6_address | default_nic | vm_type | created
>> | removed | ip6_gateway | ip6_cidr |
>>+----+--------------------------------------+-------------+---------------
>>----+---------------+-------------+-------------+---------+---------------
>>+------------+--------+--------------+----------+--------------------+----
>>----------------------------------+-----------+---------------------+-----
>>----------+-------------+-------------+--------------------+--------------
>>-------+---------------------+-------------+----------+
>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 |
>>NULL | 202 | Static | Reserved | Start |
>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 |
>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
>> | NULL |
>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 |
>>NULL | 202 | Static | Reserved | Start |
>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 |
>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
>> | NULL |
>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>> | NULL | NULL | NULL | NULL | NULL
>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 |
>>NULL | NULL | 0 | DomainRouter |
>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>> | NULL | NULL | NULL | NULL | NULL
>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 |
>>NULL | NULL | 0 | DomainRouter |
>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>> | NULL | NULL | NULL | NULL | NULL
>>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 |
>>NULL | NULL | 0 | DomainRouter |
>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>> | NULL | NULL | NULL | NULL | NULL
>>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 |
>>NULL | NULL | 0 | DomainRouter |
>>2014-03-18 08:11:32 | NULL | NULL | NULL |
>>+----+--------------------------------------+-------------+---------------
>>----+---------------+-------------+-------------+---------+---------------
>>+------------+--------+--------------+----------+--------------------+----
>>----------------------------------+-----------+---------------------+-----
>>----------+-------------+-------------+--------------------+--------------
>>-------+---------------------+-------------+----------+
>>
>>
Re: Virtual Router doesn't start
Posted by Alena Prokharchyk <Al...@citrix.com>.
The error happens not because Ip is null, but because the nic in a certain
network can¹t be found. Looks like there is some bug in VPC nic
plug/unplug for Guest networks process.
Kambiz, please do the following to fix it:
1) Stop the MS
2) Take the DB dump of cloud db in case you have to revert back.
3) Run the query:
select * from router_network_ref where router_id=<id of your VR) and
network_id not in (select network_id from nics where instance_id=<ID of
your VR> and removed is null);
It will give you the list of networks refs that somehow weren¹t cleaned
during the nic detach. Remove the entry returned from router_network_ref
table.
Let me know how it works.
-Alena.
On 3/21/14, 3:36 PM, "Kambiz Darabi" <da...@m-creations.com> wrote:
>Hello,
>
>as this is my first post to the list, I would like to thank all
>contributors for Cloudstack which I use since last fall without any
>problems. I run 4.1.1 with KVM and advanced networking.
>
>After a restart of the management server (stopping and starting the java
>process), the virtual domain router doesn't start and
>management-server.log shows a NullPointerException in
>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>
>By putting the server in debug mode and remote debugging, I found out
>that the reason is a row in the table nics which has NULL in ip (cf. row
>with id 30 in the result of the select statement below).
>
>What can I do to quickly solve this problem? Any pointers or suggestions
>are appreciated as the system is currently unusable.
>
>Thank you for your help
>
>
>Kambiz
>
>
>management-server.log:
>
>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking Ovs to prepare for
>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement]
>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated with
>any VPC
>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>Connectivity on network net1
>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the
>network id=205
>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part of
>network implement
>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Network id=202 is already implemented
>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Lock is released for network id 202 as a part of
>network implement
>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking Ovs to prepare for
>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement]
>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated
>with any VPC
>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
>Connectivity on network null
>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
>VM[DomainRouter|r-19-VM]
>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>(Job-Executor-1:job-176) No need to recreate the volume:
>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding
>disk to VM
>2014-03-18 10:03:27,224 DEBUG
>[network.router.VirtualNetworkApplianceManagerImpl]
>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0
>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1
>2014-03-18 10:03:27,343 DEBUG
>[network.router.VirtualNetworkApplianceManagerImpl]
>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>VM[DomainRouter|r-19-VM] start.
>2014-03-18 10:03:27,415 DEBUG
>[network.router.VirtualNetworkApplianceManagerImpl]
>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>balancing rules as a part of Virtual router start
>2014-03-18 10:03:27,499 DEBUG
>[network.router.VirtualNetworkApplianceManagerImpl]
>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a
>part of domR VM[DomainRouter|r-19-VM] start.
>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>(Job-Executor-1:job-176) Failed to start instance VM[DomainRouter|r-19-VM]
>java.lang.NullPointerException
> at
>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:76
>3)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetwor
>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
> at
>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNet
>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeComman
>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
> at
>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCom
>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeploy
>ment(VirtualNetworkApplianceManagerImpl.java:2205)
> at
>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerI
>mpl.java:763)
> at
>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav
>a:471)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualN
>etworkApplianceManagerImpl.java:2616)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRo
>uter(VirtualNetworkApplianceManagerImpl.java:1824)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(V
>irtualNetworkApplianceManagerImpl.java:1924)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualR
>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
> at
>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElem
>ent.java:175)
> at
>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(
>NetworkManagerImpl.java:1518)
> at
>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.j
>ava:1434)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:
>2435)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi
>rtualNetworkApplianceManagerImpl.java:2855)
> at
>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi
>rtualNetworkApplianceManagerImpl.java:2824)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(Star
>tRouterCmd.java:103)
>
>
>table nics:
>
>mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
>+----+--------------------------------------+-------------+---------------
>----+---------------+-------------+-------------+---------+---------------
>+------------+--------+--------------+----------+--------------------+----
>----------------------------------+-----------+---------------------+-----
>----------+-------------+-------------+--------------------+--------------
>-------+---------------------+-------------+----------+
>| id | uuid | instance_id | mac_address
> | ip4_address | netmask | gateway | ip_type | broadcast_uri
>| network_id | mode | state | strategy | reserver_name |
>reservation_id | device_id | update_time |
>isolation_uri | ip6_address | default_nic | vm_type | created
> | removed | ip6_gateway | ip6_cidr |
>+----+--------------------------------------+-------------+---------------
>----+---------------+-------------+-------------+---------+---------------
>+------------+--------+--------------+----------+--------------------+----
>----------------------------------+-----------+---------------------+-----
>----------+-------------+-------------+--------------------+--------------
>-------+---------------------+-------------+----------+
>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 |
>NULL | 202 | Static | Reserved | Start |
>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 |
>2014-03-17 22:21:38 | NULL | NULL | 0 |
>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL
> | NULL |
>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 |
>NULL | 202 | Static | Reserved | Start |
>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 |
>2014-03-17 22:21:38 | NULL | NULL | 0 |
>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL
> | NULL |
>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
> | NULL | NULL | NULL | NULL | NULL
>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 |
>NULL | NULL | 0 | DomainRouter |
>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL |
>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
> | NULL | NULL | NULL | NULL | NULL
>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 |
>NULL | NULL | 0 | DomainRouter |
>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL |
>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
> | NULL | NULL | NULL | NULL | NULL
>| 202 | Static | Deallocating | Start | ControlNetworkGuru |
>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 |
>NULL | NULL | 0 | DomainRouter |
>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL |
>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
> | NULL | NULL | NULL | NULL | NULL
>| 202 | Static | Allocated | Start | ControlNetworkGuru |
>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 |
>NULL | NULL | 0 | DomainRouter |
>2014-03-18 08:11:32 | NULL | NULL | NULL |
>+----+--------------------------------------+-------------+---------------
>----+---------------+-------------+-------------+---------+---------------
>+------------+--------+--------------+----------+--------------------+----
>----------------------------------+-----------+---------------------+-----
>----------+-------------+-------------+--------------------+--------------
>-------+---------------------+-------------+----------+
>
>