You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Min Chen (JIRA)" <ji...@apache.org> on 2013/07/19 03:44:48 UTC
[jira] [Resolved] (CLOUDSTACK-3498) [Object_Store_Refactor]
Deleting and adding back the zone does not create NFS Cache store and
Failure in Sytem VMs Start Up
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Chen resolved CLOUDSTACK-3498.
----------------------------------
Resolution: Fixed
> [Object_Store_Refactor] Deleting and adding back the zone does not create NFS Cache store and Failure in Sytem VMs Start Up
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3498
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3498
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Latest build from ACS 4.2 branch.
> Reporter: Sanjeev N
> Assignee: Min Chen
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: management-server.rar
>
>
> Deleting a zone does not delete the NFS Cache store in that zone and adding another zone with same NFS cache store details those were given in the deleted zone does not create another NFS Cache store in the new zone. Hence the Failure in System vms bring up.
> Steps to Reproduce:
> ================
> 1.Bring up CS with advanced zone say zone1.
> 2.Add secondary storage with S3 provider and with NFS Cache Store
> 3.Delete secondary storage
> 4.Delete zone1
> 5.Add another zone say zone2
> 6.Add secondary storage with s3 provider and NFS Cache storage in Zone2 with NFS Server and Path same as zone1
> Observations:
> ===========
> 1.Deleting zone did not delete NFS Cache store from image_store table.
> 2.Adding NFS Cache store with same details in new zone did not create another entry in image_store table for the new zone.
> Impact:
> ======
> CS downloads Routing template from S3 to NFS Cache store to bring up the System VMs. Since CS did not find NFS Cache store in the new zone to copy the routing template spinning system vms failed with following exception:
> 2013-07-12 06:28:14,731 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) needCacheStorage true, dest at null dest role Primarytemplate/tmpl/1/3/routing-3/systemvmtemplate-2013-06-25-master-kvm.qcow2.bz2 src role Image
> 2013-07-12 06:28:14,737 DEBUG [network.guru.PublicNetworkGuru] (secstorage-1:null) public network deallocate network: networkId: 210, ip: 10.147.48.3
> 2013-07-12 06:28:14,739 DEBUG [cache.allocator.StorageCacheRandomAllocator] (consoleproxy-1:null) Can't find cache storage in zone: 3
> 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) copy object failed: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 3 type ZONE
> 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) copy failed
> com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 3 type ZONE
> at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:204)
> at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:351)
> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:390)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:490)
> at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2485)
> at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-12 06:28:14,752 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 3 type ZONE
> 2013-07-12 06:28:14,752 INFO [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 3 type ZONE
> at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2493)
> at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
> at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-12 06:28:14,758 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-79-VM] in Starting state
> DB entries from Cloud table:
> ======================
> mysql> select * from image_store where role='Image' and removed is null\G;
> *************************** 1. row ***************************
> id: 3
> name: objstore
> image_provider_name: S3
> protocol: http
> url: NULL
> data_center_id: NULL
> scope: REGION
> role: Image
> uuid: 0c444304-7e5f-4137-8da1-a812c3b81cf7
> parent: NULL
> created: 2013-07-11 15:07:56
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> *************************** 2. row ***************************
> id: 5
> name: obj_store
> image_provider_name: S3
> protocol: http
> url: NULL
> data_center_id: NULL
> scope: REGION
> role: Image
> uuid: 789b6388-ea95-41e6-98e6-e2ded85f119f
> parent: NULL
> created: 2013-07-12 10:23:09
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> 2 rows in set (0.00 sec)
> mysql> select * from image_store where role='ImageCache' and removed is null\G;
> *************************** 1. row ***************************
> id: 1
> name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> image_provider_name: NFS
> protocol: nfs
> url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> data_center_id: 1
> scope: ZONE
> role: ImageCache
> uuid: 8e43fa4b-cf0c-46c2-9e68-ead6bd481742
> parent: NULL
> created: 2013-07-11 14:51:18
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> *************************** 2. row ***************************
> id: 4
> name: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
> image_provider_name: NFS
> protocol: nfs
> url: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
> data_center_id: 2
> scope: ZONE
> role: ImageCache
> uuid: b42417d4-3e97-4f75-9439-25f97badcaa1
> parent: NULL
> created: 2013-07-11 16:41:51
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> 2 rows in set (0.00 sec)
> mysql> select * from data_center;
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> | id | name | uuid | description | dns1 | dns2 | internal_dns1 | internal_dns2 | gateway | netmask | router_mac_address | mac_address | guest_network_cidr | domain | domain_id | networktype | dns_provider | gateway_provider | firewall_provider | dhcp_provider | lb_provider | vpn_provider | userdata_provider | allocation_state | zone_token | is_security_group_enabled | is_local_storage_enabled | removed | owner | created | lastUpdated | engine_state | ip6_dns1 | ip6_dns2 |
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> | 1 | Basic | d234e7a5-5cd0-4029-9b80-07a47f470e79 | NULL | 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | 02:00:00:00:00:01 | 21 | NULL | NULL | NULL | Basic | VirtualRouter | NULL | NULL | VirtualRouter | ElasticLoadBalancerVm | NULL | VirtualRouter | Enabled | 64c45795-5345-36e5-b52d-5530cd602a55 | 1 | 0 | NULL | NULL | NULL | NULL | Disabled | NULL | NULL |
> | 2 | NULL | 45072eac-10d8-499d-9325-d4ee79e45026 | NULL | 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | 02:00:00:00:00:01 | 19 | 10.1.1.0/24 | NULL | NULL | Advanced | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | Disabled | b2359ba6-39c7-3dc6-a2ca-ac9091a6a06b | 0 | 1 | 2013-07-12 10:09:14 | NULL | NULL | NULL | Disabled | NULL | NULL |
> | 3 | Adv-kvm | 37b35302-af3c-43b9-bc6c-acb17642b0d3 | NULL | 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | 02:00:00:00:00:01 | 21 | 10.1.1.0/24 | NULL | NULL | Advanced | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | Disabled | 8f2a7a84-dbfc-35e0-9201-f3066c65b414 | 0 | 1 | NULL | NULL | NULL | NULL | Disabled | NULL | NULL |
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> 3 rows in set (0.00 sec)
> 1.Data_Center 2 was removed from the setup, however we can see the image store with role image_cache for data_center 2 in the image_store table.
> 2. Data_Center 3 was created after deleting DC 2 , and provided the same image_store details as DC2, hence there is no image store with role image_cache for DC3.
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira