You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2013/07/19 13:10:49 UTC
[jira] [Created] (CLOUDSTACK-3667) [Object_Store_Refactor][VMWare]
System VMs StartCommand failed due to Exception: java.lang.Exception
Message: secondary storage for dc 3 is not ready yet?
Sanjeev N created CLOUDSTACK-3667:
-------------------------------------
Summary: [Object_Store_Refactor][VMWare] System VMs StartCommand failed due to Exception: java.lang.Exception Message: secondary storage for dc 3 is not ready yet?
Key: CLOUDSTACK-3667
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3667
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 Branch.
Zone: Advanced with VMWare cluster
Storage: S3 and ISCSI(Primary)
Reporter: Sanjeev N
Priority: Blocker
Fix For: 4.2.0
[Object_Store_Refactor][VMWare] System VMs StartCommand failed due to Exception: java.lang.Exception Message: secondary storage for dc 3 is not ready yet?
System VMs failed to start in case of VMWare cluster due to NULL in URL for S3 image store in image_store table with role Image.
I looked at the following code in incubator-cloudstack/plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java :
// prepare systemvm patch ISO
if (vmSpec.getType() != VirtualMachine.Type.User) {
// attach ISO (for patching of system VM)
String secStoreUrl = mgr.getSecondaryStorageStoreUrl(Long.parseLong(_dcId));
if(secStoreUrl == null) {
String msg = "secondary storage for dc " + _dcId + " is not ready yet?";
throw new Exception(msg);
}
mgr.prepareSecondaryStorageStore(secStoreUrl);
Since secStoreUrl is matching with null , system vms starting failing with following exception message:
java.lang.Exception: secondary storage for dc 3 is not ready yet?
Steps to Reproduce:
================
1.Bring up CS in advanced zone with vmware cluster
2.Add S3 image store and NFS cache store in the zone
3.Enable the zone
Observations:
===========
2013-07-19 11:50:10,861 WARN [vmware.resource.VmwareResource] (DirectAgent-97:10.147.40.13) StartCommand failed due to Exception: java.lang.Exception
Message: secondary storage for dc 3 is not ready yet?
java.lang.Exception: secondary storage for dc 3 is not ready yet?
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2628)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Enabled TRACE in log4j-cloud.xml and found the following sql statements executed before the exception:
mysql> SELECT image_store.id, image_store.name, image_store.uuid, image_store.protocol, image_store.url, image_store.image_provider_name, image_store.data_center_id, image_store.scope, image_store.created, image_store.removed, image_store.role, image_store.parent, image_store.total_size, image_store.used_bytes FROM image_store WHERE image_store.role = 'Image' AND (image_store.scope = 'REGION' OR image_store.data_center_id = 3 ) AND image_store.removed IS NULL;
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| id | name | uuid | protocol | url | image_provider_name | data_center_id | scope | created | removed | role | parent | total_size | used_bytes |
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| 8 | obj_store | ada9e8f2-2bf1-46b7-b19d-a9bd46a0a29d | http | NULL | S3 | NULL | REGION | 2013-07-16 12:07:09 | NULL | Image | NULL | NULL | NULL |
| 17 | obj_store2 | 9e55b78b-cead-45e4-a84e-b35896d961a8 | http | NULL | S3 | NULL | REGION | 2013-07-19 15:12:45 | NULL | Image | NULL | NULL | NULL |
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
2 rows in set (0.00 sec)
mysql> SELECT image_store.id, image_store.name, image_store.uuid, image_store.protocol, image_store.url, image_store.image_provider_name, image_store.data_center_id, image_store.scope, image_store.created, image_store.removed, image_store.role, image_store.parent, image_store.total_size, image_store.used_bytes FROM image_store WHERE image_store.id = 17 AND image_store.removed IS NULL;
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| id | name | uuid | protocol | url | image_provider_name | data_center_id | scope | created | removed | role | parent | total_size | used_bytes |
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| 17 | obj_store2 | 9e55b78b-cead-45e4-a84e-b35896d961a8 | http | NULL | S3 | NULL | REGION | 2013-07-19 15:12:45 | NULL | Image | NULL | NULL | NULL |
+----+------------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
1 row in set (0.00 sec)
mysql> SELECT image_store.id, image_store.name, image_store.uuid, image_store.protocol, image_store.url, image_store.image_provider_name, image_store.data_center_id, image_store.scope, image_store.created, image_store.removed, image_store.role, image_store.parent, image_store.total_size, image_store.used_bytes FROM image_store WHERE image_store.id = 8 AND image_store.removed IS NULL;
+----+-----------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| id | name | uuid | protocol | url | image_provider_name | data_center_id | scope | created | removed | role | parent | total_size | used_bytes |
+----+-----------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
| 8 | obj_store | ada9e8f2-2bf1-46b7-b19d-a9bd46a0a29d | http | NULL | S3 | NULL | REGION | 2013-07-16 12:07:09 | NULL | Image | NULL | NULL | NULL |
+----+-----------+--------------------------------------+----------+------+---------------------+----------------+--------+---------------------+---------+-------+--------+------------+------------+
1 row in set (0.00 sec)
ulr value is set to null incase of S3 storage provider.
Following is the image_store table content:
=================================
mysql> select * from image_store where removed is null\G;
*************************** 1. row ***************************
id: 7
name: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
image_provider_name: NFS
protocol: nfs
url: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
data_center_id: 1
scope: ZONE
role: ImageCache
uuid: aa07c444-910a-43ff-bfb7-621c0ddc6223
parent: NULL
created: 2013-07-16 12:07:09
removed: NULL
total_size: NULL
used_bytes: NULL
*************************** 2. row ***************************
id: 8
name: obj_store
image_provider_name: S3
protocol: http
url: NULL
data_center_id: NULL
scope: REGION
role: Image
uuid: ada9e8f2-2bf1-46b7-b19d-a9bd46a0a29d
parent: NULL
created: 2013-07-16 12:07:09
removed: NULL
total_size: NULL
used_bytes: NULL
*************************** 3. row ***************************
id: 16
name: nfs://10.147.28.7/export/home/sanjeev/sec_esx_os
image_provider_name: NFS
protocol: nfs
url: nfs://10.147.28.7/export/home/sanjeev/sec_esx_os
data_center_id: 3
scope: ZONE
role: ImageCache
uuid: c3e72ffb-2040-4110-856a-ffabf4ca5992
parent: NULL
created: 2013-07-19 15:12:45
removed: NULL
total_size: NULL
used_bytes: NULL
*************************** 4. row ***************************
id: 17
name: obj_store2
image_provider_name: S3
protocol: http
url: NULL
data_center_id: NULL
scope: REGION
role: Image
uuid: 9e55b78b-cead-45e4-a84e-b35896d961a8
parent: NULL
created: 2013-07-19 15:12:45
removed: NULL
total_size: NULL
used_bytes: NULL
4 rows in set (0.00 sec)
ERROR:
No query specified
Attached management server log file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira