You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2013/05/16 11:45:16 UTC
[jira] [Updated] (CLOUDSTACK-2537) [Object_Store_Refactor] System
vms failed to come up with S3 as storage provider
[ https://issues.apache.org/jira/browse/CLOUDSTACK-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjeev N updated CLOUDSTACK-2537:
----------------------------------
Attachment: management-server.rar
Attached mgmt server log file
> [Object_Store_Refactor] System vms failed to come up with S3 as storage provider
> --------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-2537
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2537
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Build from object_store branch
> Reporter: Sanjeev N
> Priority: Blocker
> Fix For: 4.2.0
>
> Attachments: management-server.rar
>
>
> System vms failed to come up with S3 as storage provider
> Repro steps:
> ==========
> 1.Bring CS in advanced zone with S3 as secondary storage provider
> 2.Enable the zone
> Expected Result:
> =============
> As and when S3 gets added to CS, it should download system template to S3 and bring up system vms.
> Actual Result:
> ===========
> System template was successfuly downloaded to S3 but failed to bring up system vms.
> Observations:
> ============
> 1.After step 1 system template was successfully downloaded to s3 bucket.
> 2013-05-16 10:40:49,434 INFO [storage.template.S3TemplateDownloader] (pool-2-thread-1:null) No credentials configured for host=10.147.28.7:80
> 2013-05-16 10:40:49,620 INFO [storage.template.S3TemplateDownloader] (pool-1-thread-1:null) Starting download from http://10.147.28.7/templates/acton/acton-systemvm-02062012.vhd.bz2 to s3 bucket imagestore remoteSize=140616708 , max size=53687091200
> 2013-05-16 10:40:49,638 DEBUG [cloud.utils.S3Utils] (pool-1-thread-1:null) Creating S3 client with configuration: [protocol: http, connectionTimeOut: 100, maxErrorRetry: 3, socketTimeout: 100]
> 2013-05-16 10:40:49,995 DEBUG [cloud.utils.S3Utils] (pool-1-thread-1:null) Setting the end point for S3 client com.amazonaws.services.s3.AmazonS3Client@470edb99 to 10.147.29.56:8080.
> 2013-05-16 10:40:58,970 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector is running...
> 2013-05-16 10:40:58,972 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) VmStatsCollector is running...
> 2013-05-16 10:40:58,974 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
> 2013-05-16 10:40:59,036 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-4:null) Seq 1-1911750666: Executing request
> 2013-05-16 10:40:59,045 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-5:null) Seq 1-1911750667: Executing request
> 2013-05-16 10:40:59,271 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-4:null) Seq 1-1911750666: Response Received:
> 2013-05-16 10:40:59,272 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-1911750666: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2013-05-16 10:40:59,356 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-5:null) Seq 1-1911750667: Response Received:
> 2013-05-16 10:40:59,357 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-1911750667: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2013-05-16 10:41:01,014 INFO [storage.template.S3TemplateDownloader] (pool-1-thread-1:null) download completed
> 2013-05-16 10:41:01,016 INFO [storage.template.DownloadManagerImpl] (pool-1-thread-1:null) Download Completion for jobId: 46e0c43d-a743-461f-8b88-16a57ac4bbb8, status=DOWNLOAD_FINISHED
> 2013-05-16 10:41:01,016 INFO [storage.template.DownloadManagerImpl] (pool-1-thread-1:null) local: template/tmpl/1/1/routing-1/acton-systemvm-02062012.vhd.bz2, bytes=140616708, error= , pct=100
> 2.After step2 CS tried to bring up the system vms. But failed to create volumes with following exceptions:
> 2013-05-16 10:43:44,843 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-1-VM]
> 2013-05-16 10:43:45,016 ERROR [storage.resource.NfsSecondaryStorageResource] (secstorage-1:null) Unable to create directory download directory 1 for download from S3.
> 2013-05-16 10:43:45,025 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Unable to create Vol[1|vm=1|ROOT]:java.lang.NullPointerException
> 2013-05-16 10:43:45,026 INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[1|vm=1|ROOT]:java.lang.NullPointerException
> at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2355)
> at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2404)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:812)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:529)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:522)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:259)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:672)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1282)
> at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-05-16 10:43:45,033 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1-VM] in Starting state
> 2013-05-16 10:43:45,036 DEBUG [agent.transport.Request] (secstorage-1:null) Seq 1-1911750674: Sending { Cmd , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"vmName":"s-1-VM","wait":0}}] }
> 2013-05-16 10:43:45,226 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
> at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:782)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:529)
> at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:522)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:259)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:672)
> at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1282)
> at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-05-16 10:43:45,227 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 1, will recycle it and start a new one
> 2013-05-16 10:43:45,236 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
> 2013-05-16 10:43:45,244 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
> mysql> select * from image_store\G;
> *************************** 1. row ***************************
> id: 1
> name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> image_provider_name: NFS
> protocol: nfs
> url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> data_center_id: 1
> scope: ZONE
> role: ImageCache
> uuid: fec804a2-c2d9-402e-a04d-1a879b0b0f30
> parent: NULL
> created: 2013-05-16 14:40:38
> removed: NULL
> total_size: NULL
> *************************** 2. row ***************************
> id: 2
> name: cb4b622e-b292-4d90-99d3-855eaa206999
> image_provider_name: S3
> protocol: http
> url: NULL
> data_center_id: NULL
> scope: REGION
> role: Image
> uuid: cb4b622e-b292-4d90-99d3-855eaa206999
> parent: NULL
> created: 2013-05-16 14:40:38
> removed: NULL
> total_size: NULL
> 2 rows in set (0.00 sec)
> ERROR:
> No query specified
> mysql> select * from storage_pool\G
> *************************** 1. row ***************************
> id: 1
> name: pri_xen_os
> uuid: c65a038a-750c-3b4f-bf26-7ce3b74e1c85
> pool_type: NetworkFilesystem
> port: 2049
> data_center_id: 1
> pod_id: 1
> cluster_id: 1
> available_bytes: 3030686924800
> capacity_bytes: 5902284816384
> host_address: 10.147.28.7
> user_info: NULL
> path: /export/home/sanjeev/pri_xen_os
> created: 2013-05-16 14:40:28
> removed: NULL
> update_time: NULL
> status: Up
> storage_provider_name: DefaultPrimary
> scope: CLUSTER
> 1 row in set (0.00 sec)
> Template status in cloud DB:
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira