You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "venkata swamybabu budumuru (JIRA)" <ji...@apache.org> on 2013/09/06 09:26:54 UTC
[jira] [Updated] (CLOUDSTACK-4621) Changing the management server's
ethernet interface / mac address leaves the system in unstable state.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
venkata swamybabu budumuru updated CLOUDSTACK-4621:
---------------------------------------------------
Attachment: logs.tgz
> Changing the management server's ethernet interface / mac address leaves the system in unstable state.
> ------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4621
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4621
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: venkata swamybabu budumuru
> Priority: Critical
> Fix For: 4.2.1
>
> Attachments: logs.tgz
>
>
> Steps to reproduce:
> 1. Have latest CloudStack setup with 4.2 build.
> 2. Have at least 1 advanced zone using Xen Cluster
> 3. deploy VMs and make sure everything works fine.
> Note : In my case, the management server deployed on VMware.
> Before MAC changes on mgmt server :
> [root@Rhel63-Sanjeev ~]# cat ifconfig.output
> eth1 Link encap:Ethernet HWaddr 06:04:5A:00:00:66
> inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
> inet6 addr: fe80::404:5aff:fe00:66/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:35035311 errors:0 dropped:0 overruns:0 frame:0
> TX packets:31941744 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:31951991629 (29.7 GiB) TX bytes:17754778160 (16.5 GiB)
> mysql> select * from mshost;
> +----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
> | id | msid | runid | name | state | version | service_ip | service_port | last_update | removed | alert_count |
> +----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
> | 1 | 6615759585382 | 1378110990284 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 04:44:45 | NULL | 0 |
> 4. I have manually logged into my vmware host and change the above MAC address to "06:04:5A:00:00:68" and that resulted in a new interface.
> [root@Rhel63-Sanjeev ~]# ifconfig
> eth2 Link encap:Ethernet HWaddr 06:04:5A:00:00:68
> inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
> inet6 addr: fe80::404:5aff:fe00:68/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:294927 errors:0 dropped:0 overruns:0 frame:0
> TX packets:475806 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:22230231 (21.2 MiB) TX bytes:392491450 (374.3 MiB)
> 5. Restart the management server and verify the cloud.mshost table
> mysql> select * from mshost;
> +----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
> | id | msid | runid | name | state | version | service_ip | service_port | last_update | removed | alert_count |
> +----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
> | 1 | 6615759585382 | 1378110990284 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 04:44:45 | NULL | 0 |
> | 2 | 6615759585384 | 1378462772622 | Rhel63-Sanjeev | Up | 4.2.0 | 10.147.59.126 | 9090 | 2013-09-06 10:20:35 | NULL | 0 |
> +----+---------------+---------------+----------------+-------+---------+---------------+--------------+---------------------+---------+-------------+
> Observations:
> i. Now it created a new entry for the mshost due to mac / interface changes
> ii. Both the above entries shows the status as UP and it assumes there are two mgmt servers
> iii. All the system VMs have the mgmt_server_id set to the old which is not infact up
> mysql> select * from host where name like '%-VM%';
> +----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
> | id | name | uuid | status | type | private_ip_address | private_netmask | private_mac_address | storage_ip_address | storage_netmask | storage_mac_address | storage_ip_address_2 | storage_mac_address_2 | storage_netmask_2 | cluster_id | public_ip_address | public_netmask | public_mac_address | proxy_port | data_center_id | pod_id | cpus | speed | url | fs_type | hypervisor_type | hypervisor_version | ram | resource | version | parent | total_size | capabilities | guid | available | setup | dom0_memory | last_ping | mgmt_server_id | disconnected | created | removed | update_count | resource_state | owner | lastUpdated | engine_state |
> +----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
> | 2 | s-1-VM | ab73e26c-7c52-4d7b-9389-97135c399ea6 | Up | SecondaryStorageVM | 10.147.40.228 | 255.255.254.0 | 06:51:08:00:00:08 | 10.147.40.228 | 255.255.254.0 | 06:51:08:00:00:08 | NULL | NULL | NULL | NULL | 10.147.48.3 | 255.255.255.0 | 06:c8:d6:00:00:0b | NULL | 1 | 1 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | s-1-VM-PremiumSecondaryStorageResource | 1 | 0 | 0 | 1346155088 | 6615759585384 | NULL | 2013-09-02 09:15:00 | NULL | 4 | Enabled | NULL | NULL | Disabled |
> | 3 | v-2-VM | c6e9f30a-b920-40f7-b320-1c6375ef287a | Up | ConsoleProxy | 10.147.40.226 | 255.255.254.0 | 06:59:6a:00:00:06 | 10.147.40.226 | 255.255.254.0 | 06:59:6a:00:00:06 | NULL | NULL | NULL | NULL | 10.147.48.4 | 255.255.255.0 | 06:35:aa:00:00:0c | NULL | 1 | 1 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | Proxy.2-ConsoleProxyResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:17:02 | NULL | 4 | Enabled | NULL | NULL | Disabled |
> | 5 | s-3-VM | 12a0b450-afe3-45b5-a88b-d528f244b016 | Up | SecondaryStorageVM | 10.147.40.240 | 255.255.254.0 | 06:04:dc:00:00:0a | 10.147.40.240 | 255.255.254.0 | 06:04:dc:00:00:0a | NULL | NULL | NULL | NULL | 10.147.48.11 | 255.255.255.0 | 06:78:00:00:00:0b | NULL | 2 | 2 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | s-3-VM-PremiumSecondaryStorageResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:44:02 | NULL | 7 | Enabled | NULL | NULL | Disabled |
> | 6 | v-4-VM | ffb8865a-d917-431b-9a61-feac7ce49b69 | Up | ConsoleProxy | 10.147.40.231 | 255.255.254.0 | 06:68:88:00:00:01 | 10.147.40.231 | 255.255.254.0 | 06:68:88:00:00:01 | NULL | NULL | NULL | NULL | 10.147.48.12 | 255.255.255.0 | 06:4b:76:00:00:0c | NULL | 2 | 2 | NULL | NULL | NoIqn | NULL | NULL | NULL | 0 | NULL | 4.2.0 | NULL | NULL | NULL | Proxy.4-ConsoleProxyResource | 1 | 0 | 0 | 1346155089 | 6615759585384 | NULL | 2013-09-02 09:46:14 | NULL | 4 | Enabled | NULL | NULL | Disabled |
> +----+--------+--------------------------------------+--------+--------------------+--------------------+-----------------+---------------------+--------------------+-----------------+---------------------+----------------------+-----------------------+-------------------+------------+-------------------+----------------+--------------------+------------+----------------+--------+------+-------+-------+---------+-----------------+--------------------+------+----------+---------+--------+------------+--------------+----------------------------------------+-----------+-------+-------------+------------+----------------+--------------+---------------------+---------+--------------+----------------+-------+-------------+--------------+
> iv. This resulted in issues like unable to deploy user VMs and also unable restart/destroy system VMs
> 2013-09-06 01:11:31,295 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-873922573: Forwarding Seq 1-873922573: { Cmd , MgmtId: 6615759585384, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"fabe8c72-74a8-4816-80f5-64fb7a4de583","origUrl":"http://10.147.28.7/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2","uuid":"0bf2c102-13aa-11e3-a816-06045a000066","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1f5d583d-ef0b-4689-a01d-2848dfabdead","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-20","size":223003018,"volumeId":27,"vmName":"r-20-VM","accountId":2,"format":"VHD","id":27,"hypervisorType":"XenServer"}},"executeInSequence":true,"wait":0}}] } to 6615759585382
> Attaching all the relevant logs along with db dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira