You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "venkata swamybabu budumuru (JIRA)" <ji...@apache.org> on 2013/09/06 08:56:54 UTC
[jira] [Created] (CLOUDSTACK-4621) Changing the management server's
ethernet interface / mac address leaves the system in unstable state.
venkata swamybabu budumuru created CLOUDSTACK-4621:
------------------------------------------------------
Summary: Changing the management server's ethernet interface / mac address leaves the system in unstable state.
Key: CLOUDSTACK-4621
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4621
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: venkata swamybabu budumuru
Priority: Critical
Fix For: 4.2.1
Steps to reproduce:
1. Have latest CloudStack setup with 4.2 build.
2. Have at least 1 advanced zone using Xen Cluster
3. deploy VMs and make sure everything works fine.
Note : In my case, the management server deployed on VMware.
Before MAC changes on mgmt server :
[root@Rhel63-Sanjeev ~]# cat ifconfig.output
eth1 Link encap:Ethernet HWaddr 06:04:5A:00:00:66
inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
inet6 addr: fe80::404:5aff:fe00:66/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:35035311 errors:0 dropped:0 overruns:0 frame:0
TX packets:31941744 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:31951991629 (29.7 GiB) TX bytes:17754778160 (16.5 GiB)
mysql> select * from mshost;
+----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
| id | msid | runid | name | state | version | service_ip | service_port | last_update | removed | alert_count |
+----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
| 1 | 6615759585382 | 1378110990284 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 04:44:45 | NULL | 0 |
4. I have manually logged into my vmware host and change the above MAC address to "06:04:5A:00:00:68" and that resulted in a new interface.
[root@Rhel63-Sanjeev ~]# ifconfig
eth2 Link encap:Ethernet HWaddr 06:04:5A:00:00:68
inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
inet6 addr: fe80::404:5aff:fe00:68/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:294927 errors:0 dropped:0 overruns:0 frame:0
TX packets:475806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22230231 (21.2 MiB) TX bytes:392491450 (374.3 MiB)
5. Restart the management server and verify the cloud.mshost table
mysql> select * from mshost;
+----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
| id | msid | runid | name | state | version | service_ip | service_port | last_update | removed | alert_count |
+----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
| 1 | 6615759585382 | 1378110990284 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 04:44:45 | NULL | 0 |
| 2 | 6615759585384 | 1378462772622 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 10:20:35 | NULL | 0 |
+----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
Observations:
i. Now it created a new entry for the mshost due to mac / interface changes
ii. Both the above entries shows the status as UP and it assumes there are two mgmt servers
iii. All the system VMs have the mgmt_server_id set to the old which is not infact up
mysql> select * from host where name like '%-VM%';
+----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
| id | name | uuid | status | type | private_ip_address | private_netmask | private_mac_address | storage_ip_address | storage_netmask | storage_mac_address | storage_ip_address_2 | storage_mac_address_2 | storage_netmask_2 | cluster_id | public_ip_address | public_netmask | public_mac_address | proxy_port | data_center_id | pod_id | cpus | speed | url | fs_type | hypervisor_type | hypervisor_version | ram | resource | version | parent | total_size | capabilities | guid | available | setup | dom0_memory | last_ping | mgmt_server_id | disconnected | created | removed | update_count | resource_state | owner | lastUpdated | engine_state |
+----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
| 2 | s-1-VM | ab73e26c-7c52-4d7b-9389-97135c399ea6 | Up | SecondaryStorageVM | 10.147.40.228 | 255.255.254.0 | 06:51:08:00:00:08 | 10.147.40.228 | 255.255.254.0 | 06:51:08:00:00:08 | NULL | NULL | NULL | NULL | 10.147.48.3 | 255.255.255.0 | 06:c8:d6:00:00:0b | NULL | 1 | 1 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | s-1-VM-PremiumSecondaryStorageResource | 1 | 0 | 0 | 1346155088 | 6615759585384 | NULL | 2013-09-02 09:15:00 | NULL | 4 | Enabled | NULL | NULL | Disabled |
| 3 | v-2-VM | c6e9f30a-b920-40f7-b320-1c6375ef287a | Up | ConsoleProxy | 10.147.40.226 | 255.255.254.0 | 06:59:6a:00:00:06 | 10.147.40.226 | 255.255.254.0 | 06:59:6a:00:00:06 | NULL | NULL | NULL | NULL | 10.147.48.4 | 255.255.255.0 | 06:35:aa:00:00:0c | NULL | 1 | 1 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | Proxy.2-ConsoleProxyResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:17:02 | NULL | 4 | Enabled | NULL | NULL | Disabled |
| 5 | s-3-VM | 12a0b450-afe3-45b5-a88b-d528f244b016 | Up | SecondaryStorageVM | 10.147.40.240 | 255.255.254.0 | 06:04:dc:00:00:0a | 10.147.40.240 | 255.255.254.0 | 06:04:dc:00:00:0a | NULL | NULL | NULL | NULL | 10.147.48.11 | 255.255.255.0 | 06:78:00:00:00:0b | NULL | 2 | 2 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | s-3-VM-PremiumSecondaryStorageResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:44:02 | NULL | 7 | Enabled | NULL | NULL | Disabled |
| 6 | v-4-VM | ffb8865a-d917-431b-9a61-feac7ce49b69 | Up | ConsoleProxy | 10.147.40.231 | 255.255.254.0 | 06:68:88:00:00:01 | 10.147.40.231 | 255.255.254.0 | 06:68:88:00:00:01 | NULL | NULL | NULL | NULL | 10.147.48.12 | 255.255.255.0 | 06:4b:76:00:00:0c | NULL | 2 | 2 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | Proxy.4-ConsoleProxyResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:46:14 | NULL | 4 | Enabled | NULL | NULL | Disabled |
+----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
iv. This resulted in issues like unable to deploy user VMs and also unable restart/destroy system VMs
2013-09-06 01:11:31,295 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-873922573: Forwarding Seq 1-873922573: { Cmd , MgmtId: 6615759585384, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"fabe8c72-74a8-4816-80f5-64fb7a4de583","origUrl":"http://10.147.28.7/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2","uuid":"0bf2c102-13aa-11e3-a816-06045a000066","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1f5d583d-ef0b-4689-a01d-2848dfabdead","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-20","size":223003018,"volumeId":27,"vmName":"r-20-VM","accountId":2,"format":"VHD","id":27,"hypervisorType":"XenServer"}},"executeInSequence":true,"wait":0}}] } to 6615759585382
Attaching all the relevant logs along with db dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira