You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Andrey Padiy (JIRA)" <ji...@apache.org> on 2013/05/01 11:08:17 UTC
[jira] [Comment Edited] (CLOUDSTACK-528) Failed to create secondary
storage
[ https://issues.apache.org/jira/browse/CLOUDSTACK-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646474#comment-13646474 ]
Andrey Padiy edited comment on CLOUDSTACK-528 at 5/1/13 9:07 AM:
-----------------------------------------------------------------
I am experiencing exactly the same problem. Getting "sent incorrect data center: null " stack trace and unable to add secondary storage.
* Upgraded from CS 3.0.2 to 4.0.2 yesterday morning.
* When running "cloud-sysvmadm" post upgrade to restart system VMs I received I/O device errors for both SSVM and Console Proxy VM (In XenCenter console) and system VMs wouldn't start. (Router VM restarted OK)
* Destroyed SSVM and Console Proxy VMs (through CS panel) hoping they will get recreated.
* Waited for few hours - nothing happened.
* Redownloaded systemVM template.
* Detached secondary storage
* Tried attaching secondary storage and now getting stack trace
INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) Trying to add a new host at nfs://10.0.0.126/mnt/plestore02-nas/plestore02-cloudstack-secondary/CSSecondary in data center 1
WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Wait, what's going on? eth1ip is null!!
WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Wait, there is no storageip in /proc/cmdline, something wrong!
ERROR [cloud.resource.ServerResourceBase] (catalina-exec-2:) Nics are not configured!
INFO [cloud.resource.ServerResourceBase] (catalina-exec-2:) Designating private to be nic eth0
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script config_ssl.sh
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script config_auth.sh
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script ipfirewall.sh
INFO [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Secondary storage runs in role templateProcessor
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) listvmtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) createtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) listvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VhdProcessor - VHD Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.IsoProcessor - ISO Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VmdkProcessor - VMDK Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.RawImageProcessor - Raw Image Processor
WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.40.0.20 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:544)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:423)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:312)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:64)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) server resources successfully discovered by SecondaryStorage
WARN [cloud.api.ApiDispatcher] (catalina-exec-2:) class com.cloud.api.ServerApiException : Failed to add secondary storage
I then attached to cloudstack-management process using strace and noticed few things that happen during secondary storage attachment process :
* It tries creating a symlink from "/mnt/SecStorage" to "/var/www/html/copy" on the management server. - I don't have Apache installed on the box so it failed and I had to manually create "/var/www/html/copy" directory and give "cloud" user perms
*After a few SELECTS it then runs the following SQL query which I suspect the route cause of the stack trace:
SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled FROM data_center WHERE data_center.name IS NULL AND data_center.removed IS NULL ORDER BY RAND() LIMIT 1
This query returns nothing because of the "WHERE data_center.name IS NULL" and I suspect that's why I am getting "sent incorrect data center: null "
I do have a row inside "data_center" table and the name is definitely not NULL.
mysql> SELECT * FROM data_center\G;
*************************** 1. row ***************************
id: 1
name: OurCompanyName
uuid: fa31e8f8-3ad9-46ce-8813-71819d672fe6
description: NULL
dns1: 10.0.0.250
dns2: 10.40.0.240
internal_dns1: 10.0.0.250
internal_dns2: 10.40.0.240
gateway: NULL
netmask: NULL
router_mac_address: 02:00:00:00:00:01
mac_address: 453
guest_network_cidr: NULL
domain: OurDomain.com
domain_id: NULL
networktype: Basic
dns_provider: VirtualRouter
gateway_provider: NULL
firewall_provider: NULL
dhcp_provider: VirtualRouter
lb_provider: ElasticLoadBalancerVm
vpn_provider: NULL
userdata_provider: VirtualRouter
allocation_state: Enabled
zone_token: fb7e650b-ab22-3797-b1b3-5a54e5c9bf07
is_security_group_enabled: 1
removed: NULL
is_local_storage_enabled: 0
1 row in set (0.00 sec)
Can someone confirm whether that query should contain "WHERE data_center.name IS NULL" and if not how do I change it? I am guessing one of the jar files need to be recompiled. Is there quick fix for this?
I need to upload new ISOs and create new templates and have no means of doing it without secondary storage.
Thanks
was (Author: hyperbole):
I am experiencing exactly the same problem. Getting "sent incorrect data center: null " stack trace and unable to add secondary storage.
* Upgraded from CS 3.0.2 to 4.0.2 yesterday morning.
* When running "cloud-sysvmadm" post upgrade to restart system VMs I received I/O device errors for both SSVM and Console Proxy VM (In XenCenter console) and system VMs wouldn't start. (Router VM restarted OK)
* Destroyed SSVM and Console Proxy VMs (through CS panel) hoping they will get recreated.
* Waited for few hours - nothing happened.
* Redownloaded systemVM template.
* Detached secondary storage
* Tried attaching secondary storage and now getting stack trace
{noformat}
INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) Trying to add a new host at nfs://10.0.0.126/mnt/plestore02-nas/plestore02-cloudstack-secondary/CSSecondary in data center 1
WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Wait, what's going on? eth1ip is null!!
WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Wait, there is no storageip in /proc/cmdline, something wrong!
ERROR [cloud.resource.ServerResourceBase] (catalina-exec-2:) Nics are not configured!
INFO [cloud.resource.ServerResourceBase] (catalina-exec-2:) Designating private to be nic eth0
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script config_ssl.sh
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script config_auth.sh
WARN [utils.script.Script] (catalina-exec-2:) Unable to find script ipfirewall.sh
INFO [storage.resource.NfsSecondaryStorageResource] (catalina-exec-2:) Secondary storage runs in role templateProcessor
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) listvmtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) createtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) listvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
INFO [storage.template.DownloadManagerImpl] (catalina-exec-2:) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VhdProcessor - VHD Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.IsoProcessor - ISO Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VmdkProcessor - VMDK Processor
INFO [utils.component.ComponentLocator] (catalina-exec-2:) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.RawImageProcessor - Raw Image Processor
WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.40.0.20 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:544)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:423)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:312)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:64)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-2:) server resources successfully discovered by SecondaryStorage
WARN [cloud.api.ApiDispatcher] (catalina-exec-2:) class com.cloud.api.ServerApiException : Failed to add secondary storage
{noformat}
I then attached to cloudstack-management process using strace and noticed few things that happen during secondary storage attachment process :
* It tries creating a symlink from "/mnt/SecStorage" to "/var/www/html/copy" on the management server. - I don't have Apache installed on the box so it failed and I had to manually create "/var/www/html/copy" directory and give "cloud" user perms
*After a few SELECTS it then runs the following SQL query which I suspect the route cause of the stack trace:
{noformat}
SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled FROM data_center WHERE data_center.name IS NULL AND data_center.removed IS NULL ORDER BY RAND() LIMIT 1
{noformat}
This query returns nothing because of the "WHERE data_center.name IS NULL" and I suspect that's why I am getting "sent incorrect data center: null "
I do have a row inside "data_center" table and the name is definitely not NULL.
{noformat}
mysql> SELECT * FROM data_center\G;
*************************** 1. row ***************************
id: 1
name: OurCompanyName
uuid: fa31e8f8-3ad9-46ce-8813-71819d672fe6
description: NULL
dns1: 10.0.0.250
dns2: 10.40.0.240
internal_dns1: 10.0.0.250
internal_dns2: 10.40.0.240
gateway: NULL
netmask: NULL
router_mac_address: 02:00:00:00:00:01
mac_address: 453
guest_network_cidr: NULL
domain: OurDomain.com
domain_id: NULL
networktype: Basic
dns_provider: VirtualRouter
gateway_provider: NULL
firewall_provider: NULL
dhcp_provider: VirtualRouter
lb_provider: ElasticLoadBalancerVm
vpn_provider: NULL
userdata_provider: VirtualRouter
allocation_state: Enabled
zone_token: fb7e650b-ab22-3797-b1b3-5a54e5c9bf07
is_security_group_enabled: 1
removed: NULL
is_local_storage_enabled: 0
1 row in set (0.00 sec)
{noformat}
Can someone confirm whether that query should contain "WHERE data_center.name IS NULL" and if not how do I change it? I am guessing one of the jar files need to be recompiled. Is there quick fix for this?
I need to upload new ISOs and create new templates and have no means of doing it without secondary storage.
Thanks
> Failed to create secondary storage
> ----------------------------------
>
> Key: CLOUDSTACK-528
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-528
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Storage Controller
> Affects Versions: 4.0.0
> Environment: CentOS
> Reporter: Tamas Monos
> Priority: Blocker
> Fix For: 4.0.2
>
>
> Hi,
> I cannot add secondary storage to my existing Zone. This prevents the platform from operating.
> API call via GUI captured by tcpdump:
> http://sandbox.veber.co.uk/client/api?command=addSecondaryStorage&zoneId=a8d21e6d-bc9e-4dc2-b996-fc22727cd9c1&url=nfs%3A%2F%2F192.168.1.2%2Fwatford_cloud&response=json&sessionkey=DGl3OmkfQRGwuIDZFoOpoF2WBJU%3D&_=1353673985114
> Result:
> Unable to connect due to
> java.lang.IllegalArgumentException: Host 192.168.1.4 sent incorrect data center: null
> at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
> at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
> at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
> at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2012-11-23 12:32:58,518 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-23:null) server resources successfully discovered by SecondaryStorage
> 2012-11-23 12:32:58,518 WARN [cloud.api.ApiDispatcher] (catalina-exec-23:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
> Debug:
> mysql> select uuid from data_center where name='Watford';
> +--------------------------------------+
> | uuid |
> +--------------------------------------+
> | a8d21e6d-bc9e-4dc2-b996-fc22727cd9c1 |
> +--------------------------------------+
> The Zone ID matches but API throws exception while the IS is supplied in the query.
> Scenario:
> I have upgraded from 3.0.2 to 4.0.0. Due to the upgrade scripts did not update the systemVMs to the new template I've applied a workaround and recreated the systemVMs with the new templates. After that I've enabled storage clean-up and the cleanup script destroyed everything on my primary storage including systemVMs. I wanted to re-create them but cannot add secondary storage to the Zone.
> The applied workarounds during upgrade:
> 1. Add new vmware template to CS with name systemvm-vmware-4.0 (I have re-imported it) 2. Wait till the template is downloaded and installed successfully
> 3. Look up id of this template in DB (Name should match the input provided in step
> mysql> select id from `cloud`.`vm_template` where name =
> mysql> 'systemvm-vmware-4.0' and removed is null;
> 4. Update template type to SYSTEM
> mysql> update `cloud`.`vm_template` set type='SYSTEM' where id =
> mysql> <id-from-step3>;
> 5. Update template Id for all system Vms
> mysql> update `cloud`.`vm_instance` set vm_template_id = <id-from-step3>
> mysql> where type <> 'User' and hypervisor_type = 'VMware';
> 6. Restart all system Vms
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira