You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ben Linton <be...@hivelocity.net> on 2013/01/24 20:21:12 UTC
Issue with secondary storage on Cloudstack 4 - Host sent incorrect data center: null
Hello,
I am having a strange problem adding secondary storage to our cloudstack system. I've had the same secondary storage added in the past and I only recently removed while doing some clean up and repair. However now we cannot add it back. I am using the management server for secondary storage and I have cleared out the directory and re-downloaded the xen system vm template image. I am using the private IP address of the management server and /export/secondary when trying to add secondary storage
Once I try to add it fails immediately, I will post the full log at the bottom but these are some entries that could pertain to why it is failing --
Once I hit "okay these are the following entries -
2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
Then it searches for a bunch of scripts and we arrive at this point and fails totally
cessor
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
For a little background -
Our cloudstack system consists of one management server and 2 xen hypervisors setup using advanced networking and multiple primary storage targets. This was a working system on 3.0.2(and prior versions).
Our latest issue stemmed from an issue with a failed primary storage iscsi target. Our system VMs were trying to boot using this target and we had quite a time removing this old failed target. During this process we upgraded to 4.x to try to use the "Force" option on primary storage. I eventually removed the primary storage entry from the database using a procedure I found online. After that the system VMs weren't trying to boot off of that target however there was some issues with secondary storage so I tried to remove and re-add as this fixed a similar issue we had in the past. However now we can't add storage back and I've search and tried many different things to no avail.
I am about ready to dump the entire system and reload. I would rather like to get this issue resolved as we are going to build a production cloudstack and it would be nice to see we don't encounter issues that we can't resolve.
If there are other details needed let me know. Any help or advice is certainly appreciated.
2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in the classpath
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././config_ssl.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/./config_ssl.sh
2013-01-24 13:51:44,916 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_ssl.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in the classpath
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/./config_auth.sh
2013-01-24 13:51:44,917 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in the classpath
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/./ipfirewall.sh
2013-01-24 13:51:44,919 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script ipfirewall.sh
2013-01-24 13:51:44,919 INFO [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Secondary storage runs in role templateProcessor
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in the classpath
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,923 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvmtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in the classpath
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in the classpath
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,925 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in the classpath
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,926 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VhdProcessor - VHD Processor
2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.IsoProcessor - ISO Processor
2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VmdkProcessor - VMDK Processor
2013-01-24 13:51:44,929 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.RawImageProcessor - Raw Image Processor
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-01-24 13:51:44,954 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) server resources successfully discovered by SecondaryStorage
2013-01-24 13:51:44,954 WARN [cloud.api.ApiDispatcher] (catalina-exec-10:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
2013-01-24 13:51:55,168 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
Best regards,
Ben Linton
Network Operations
Hivelocity Ventures Corp.
http://www.hivelocity.net
1-888-869(HOST)4678 x 222
Fax (813) 902-6477
Skype/AIM - HivelocityBen
MSN - ben@hivelocity.net
RE: Issue with secondary storage on Cloudstack 4 - Host sent
incorrect data center: null
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
Yes I reported a bug on the 23/11/2012.
See here: https://issues.apache.org/jira/browse/CLOUDSTACK-528
Please see related bugs 531,532 as well.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk
Follow us on Twitter: www.twitter.com/veberhost
Follow us on Facebook: www.facebook.com/veberhost
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: 28 January 2013 20:53
To: cloudstack-users@incubator.apache.org
Subject: Re: Issue with secondary storage on Cloudstack 4 - Host sent incorrect data center: null
Hi Tamas, did you file a bug for this ?
We need to figure out if it's really a bug in 4.0 or not . 4.0 is supposed to be a stable release that we can use in production. You or Ben should not have to re-vert back to 3.0.2.
Ahmad, maybe you can help here ?
thanks,
-Sebastien
On Jan 25, 2013, at 2:00 PM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I had exactly the same problem, and the only thing saved me in my test environment the fact I was running the management and database servers on virtual so I had a snapshot to revert to.
> After upgrade from 3.0.2 to 4.0.0 my CS jumped off the cliff, systemVM templates self-destructed, could not remove/add secondary storage because of the same error: "Host sent incorrect data center: null"
>
> The only thing I can suggest is to ditch your current install, build a 3.0.2 and disable storage clean-up in global config as it can result in killing your snapshots and remove actually required templates.
> Apart from that 3.0.2 seems solid, in production with it for about a year now, just works, everyone is happy.
> I've have also hacked the 4.0 console-proxy back into the 3.0.2 systemISO so it follows resolution changes in 3.0.2 too.
>
> After going into production make sure you have a test environment for testing upgrades and new features.
>
> Regards
>
> Tamas Monos DDI +44(0)2034687012
> Chief Technical Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522 7057
> http://www.veber.co.uk
>
> Follow us on Twitter: www.twitter.com/veberhost Follow us on Facebook:
> www.facebook.com/veberhost
>
> -----Original Message-----
> From: Ben Linton [mailto:ben@hivelocity.net]
> Sent: 24 January 2013 19:21
> To: cloudstack-users@incubator.apache.org
> Subject: Issue with secondary storage on Cloudstack 4 - Host sent
> incorrect data center: null
>
> Hello,
>
>
> I am having a strange problem adding secondary storage to our cloudstack system. I've had the same secondary storage added in the past and I only recently removed while doing some clean up and repair. However now we cannot add it back. I am using the management server for secondary storage and I have cleared out the directory and re-downloaded the xen system vm template image. I am using the private IP address of the management server and /export/secondary when trying to add secondary storage
>
> Once I try to add it fails immediately, I will post the full log at
> the bottom but these are some entries that could pertain to why it is
> failing --
>
> Once I hit "okay these are the following entries -
>
> 2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl]
> (catalina-exec-10:null) Trying to add a new host at
> nfs://10.0.0.15/export/secondary in data center 1
> 2013-01-24 13:51:44,830 DEBUG
> [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null)
> Executing: sudo mount 10.0.0.15:/export/secondary
> /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,871 DEBUG
> [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null)
> Executing: sudo umount /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
> 2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Unable to get network interface for xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Unable to get network interface for xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
> 2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Designating private to be nic eth1
>
> Then it searches for a bunch of scripts and we arrive at this point
> and fails totally
>
> cessor
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 -
> b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 -
> b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG
> [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null)
> Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,945 DEBUG [agent.transport.Request]
> (catalina-exec-10:null) Seq -1--1: Startup request from directly
> connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111,
> [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.1
> 5","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.2
> 55.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0",
> "storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
> 2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl]
> (catalina-exec-10:null) Unable to connect due to
> java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
> at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
> at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
> at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
> at
> com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStor
> ageCmd.java:79)
>
>
>
> For a little background -
>
> Our cloudstack system consists of one management server and 2 xen hypervisors setup using advanced networking and multiple primary storage targets. This was a working system on 3.0.2(and prior versions).
>
> Our latest issue stemmed from an issue with a failed primary storage iscsi target. Our system VMs were trying to boot using this target and we had quite a time removing this old failed target. During this process we upgraded to 4.x to try to use the "Force" option on primary storage. I eventually removed the primary storage entry from the database using a procedure I found online. After that the system VMs weren't trying to boot off of that target however there was some issues with secondary storage so I tried to remove and re-add as this fixed a similar issue we had in the past. However now we can't add storage back and I've search and tried many different things to no avail.
>
> I am about ready to dump the entire system and reload. I would rather like to get this issue resolved as we are going to build a production cloudstack and it would be nice to see we don't encounter issues that we can't resolve.
>
>
> If there are other details needed let me know. Any help or advice is certainly appreciated.
>
>
>
>
>
>
> 2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl]
> (catalina-exec-10:null) Trying to add a new host at
> nfs://10.0.0.15/export/secondary in data center 1
> 2013-01-24 13:51:44,830 DEBUG
> [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null)
> Executing: sudo mount 10.0.0.15:/export/secondary
> /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,871 DEBUG
> [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null)
> Executing: sudo umount /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
> 2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Unable to get network interface for xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Unable to get network interface for xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
> 2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Designating private to be nic eth1
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in the classpath
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/java/./scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/java/./scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/java/./scripts/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/java/./config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/java/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/share/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/common/./scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/common/./scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/common/./scripts/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/common/./config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/common/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/cloud/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/lib64/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /usr/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /var/lib/cloud/management/././scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /var/lib/cloud/management/././scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /var/lib/cloud/management/././scripts/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /var/lib/cloud/management/././config_ssl.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_ssl.sh in
> /var/lib/cloud/management/./config_ssl.sh
> 2013-01-24 13:51:44,916 WARN [utils.script.Script]
> (catalina-exec-10:null) Unable to find script config_ssl.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in the classpath
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/java/./scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/java/./scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/java/./scripts/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/java/./config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/java/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/share/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/common/./scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/common/./scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/common/./scripts/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/common/./config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/common/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/cloud/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/lib64/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /usr/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /var/lib/cloud/management/././scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /var/lib/cloud/management/././scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /var/lib/cloud/management/././scripts/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /var/lib/cloud/management/././config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for config_auth.sh in
> /var/lib/cloud/management/./config_auth.sh
> 2013-01-24 13:51:44,917 WARN [utils.script.Script]
> (catalina-exec-10:null) Unable to find script config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in the classpath
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/java/./scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/java/./scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/java/./scripts/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/java/./ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/java/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/share/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/common/./scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/common/./scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/common/./scripts/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/common/./ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/common/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/cloud/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/lib64/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /usr/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /var/lib/cloud/management/././scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /var/lib/cloud/management/././scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /var/lib/cloud/management/././scripts/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /var/lib/cloud/management/././ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for ipfirewall.sh in
> /var/lib/cloud/management/./ipfirewall.sh
> 2013-01-24 13:51:44,919 WARN [utils.script.Script]
> (catalina-exec-10:null) Unable to find script ipfirewall.sh
> 2013-01-24 13:51:44,919 INFO
> [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null)
> Secondary storage runs in role templateProcessor
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in the classpath
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/share/java/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/share/java/scripts/storage/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/share/java/scripts/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/share/java/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/share/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in /listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvmtmplt.sh in
> /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,923 INFO [storage.template.DownloadManagerImpl]
> (catalina-exec-10:null) listvmtmplt.sh found in
> /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in the classpath
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/share/java/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/share/java/scripts/storage/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/share/java/scripts/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/share/java/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/share/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in /createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createtmplt.sh in
> /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 INFO [storage.template.DownloadManagerImpl]
> (catalina-exec-10:null) createtmplt.sh found in
> /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in the classpath
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/share/java/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/share/java/scripts/storage/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/share/java/scripts/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/share/java/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/share/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in /listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for listvolume.sh in
> /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,925 INFO [storage.template.DownloadManagerImpl]
> (catalina-exec-10:null) listvolume.sh found in
> /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in the classpath
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/share/java/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/share/java/scripts/storage/createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/share/java/scripts/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/share/java/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/share/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) environment.properties says scripts should be
> in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script]
> (catalina-exec-10:null) Looking for createvolume.sh in
> /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,926 INFO [storage.template.DownloadManagerImpl]
> (catalina-exec-10:null) createvolume.sh found in
> /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator]
> (catalina-exec-10:null) Found component:
> com.cloud.storage.template.Processor in
> com.cloud.storage.template.VhdProcessor - VHD Processor
> 2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator]
> (catalina-exec-10:null) Found component:
> com.cloud.storage.template.Processor in
> com.cloud.storage.template.IsoProcessor - ISO Processor
> 2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator]
> (catalina-exec-10:null) Found component:
> com.cloud.storage.template.Processor in
> com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
> 2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator]
> (catalina-exec-10:null) Found component:
> com.cloud.storage.template.Processor in
> com.cloud.storage.template.VmdkProcessor - VMDK Processor
> 2013-01-24 13:51:44,929 INFO [utils.component.ComponentLocator]
> (catalina-exec-10:null) Found component:
> com.cloud.storage.template.Processor in
> com.cloud.storage.template.RawImageProcessor - Raw Image Processor
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 -
> b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase]
> (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 -
> b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG
> [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null)
> Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,945 DEBUG [agent.transport.Request]
> (catalina-exec-10:null) Seq -1--1: Startup request from directly
> connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111,
> [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.1
> 5","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.2
> 55.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0",
> "storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
> 2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl]
> (catalina-exec-10:null) Unable to connect due to
> java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
> at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
> at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
> at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
> at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-01-24 13:51:44,954 INFO [cloud.resource.ResourceManagerImpl]
> (catalina-exec-10:null) server resources successfully discovered by
> SecondaryStorage
> 2013-01-24 13:51:44,954 WARN [cloud.api.ApiDispatcher]
> (catalina-exec-10:null) class com.cloud.api.ServerApiException :
> Failed to add secondary storage
> 2013-01-24 13:51:55,168 DEBUG
> [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null)
> Zone 1 is not ready to launch console proxy yet
>
>
> Best regards,
>
> Ben Linton
> Network Operations
> Hivelocity Ventures Corp.
> http://www.hivelocity.net
> 1-888-869(HOST)4678 x 222
> Fax (813) 902-6477
> Skype/AIM - HivelocityBen
> MSN - ben@hivelocity.net
>
>
Re: Issue with secondary storage on Cloudstack 4 - Host sent incorrect data center: null
Posted by Sebastien Goasguen <ru...@gmail.com>.
Hi Tamas, did you file a bug for this ?
We need to figure out if it's really a bug in 4.0 or not . 4.0 is supposed to be a stable release that we can use in production. You or Ben should not have to re-vert back to 3.0.2.
Ahmad, maybe you can help here ?
thanks,
-Sebastien
On Jan 25, 2013, at 2:00 PM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I had exactly the same problem, and the only thing saved me in my test environment the fact I was running the management and database servers on virtual so I had a snapshot to revert to.
> After upgrade from 3.0.2 to 4.0.0 my CS jumped off the cliff, systemVM templates self-destructed, could not remove/add secondary storage because of the same error: "Host sent incorrect data center: null"
>
> The only thing I can suggest is to ditch your current install, build a 3.0.2 and disable storage clean-up in global config as it can result in killing your snapshots and remove actually required templates.
> Apart from that 3.0.2 seems solid, in production with it for about a year now, just works, everyone is happy.
> I've have also hacked the 4.0 console-proxy back into the 3.0.2 systemISO so it follows resolution changes in 3.0.2 too.
>
> After going into production make sure you have a test environment for testing upgrades and new features.
>
> Regards
>
> Tamas Monos DDI +44(0)2034687012
> Chief Technical Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522 7057
> http://www.veber.co.uk
>
> Follow us on Twitter: www.twitter.com/veberhost
> Follow us on Facebook: www.facebook.com/veberhost
>
> -----Original Message-----
> From: Ben Linton [mailto:ben@hivelocity.net]
> Sent: 24 January 2013 19:21
> To: cloudstack-users@incubator.apache.org
> Subject: Issue with secondary storage on Cloudstack 4 - Host sent incorrect data center: null
>
> Hello,
>
>
> I am having a strange problem adding secondary storage to our cloudstack system. I've had the same secondary storage added in the past and I only recently removed while doing some clean up and repair. However now we cannot add it back. I am using the management server for secondary storage and I have cleared out the directory and re-downloaded the xen system vm template image. I am using the private IP address of the management server and /export/secondary when trying to add secondary storage
>
> Once I try to add it fails immediately, I will post the full log at the bottom but these are some entries that could pertain to why it is failing --
>
> Once I hit "okay these are the following entries -
>
> 2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
> 2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
> 2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
> 2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
>
> Then it searches for a bunch of scripts and we arrive at this point and fails totally
>
> cessor
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
> 2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
> java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
> at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
> at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
> at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
> at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
>
>
>
> For a little background -
>
> Our cloudstack system consists of one management server and 2 xen hypervisors setup using advanced networking and multiple primary storage targets. This was a working system on 3.0.2(and prior versions).
>
> Our latest issue stemmed from an issue with a failed primary storage iscsi target. Our system VMs were trying to boot using this target and we had quite a time removing this old failed target. During this process we upgraded to 4.x to try to use the "Force" option on primary storage. I eventually removed the primary storage entry from the database using a procedure I found online. After that the system VMs weren't trying to boot off of that target however there was some issues with secondary storage so I tried to remove and re-add as this fixed a similar issue we had in the past. However now we can't add storage back and I've search and tried many different things to no avail.
>
> I am about ready to dump the entire system and reload. I would rather like to get this issue resolved as we are going to build a production cloudstack and it would be nice to see we don't encounter issues that we can't resolve.
>
>
> If there are other details needed let me know. Any help or advice is certainly appreciated.
>
>
>
>
>
>
> 2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
> 2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
> 2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
> 2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
> 2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
> 2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
> 2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in the classpath
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/config_ssl.sh
> 2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/config_ssl.sh
> 2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././config_ssl.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/./config_ssl.sh
> 2013-01-24 13:51:44,916 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_ssl.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in the classpath
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
> 2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/./config_auth.sh
> 2013-01-24 13:51:44,917 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_auth.sh
> 2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in the classpath
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./ipfirewall.sh
> 2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/secondary/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././ipfirewall.sh
> 2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/./ipfirewall.sh
> 2013-01-24 13:51:44,919 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script ipfirewall.sh
> 2013-01-24 13:51:44,919 INFO [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Secondary storage runs in role templateProcessor
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in the classpath
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,923 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvmtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in the classpath
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in the classpath
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,925 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in the classpath
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/createvolume.sh
> 2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /createvolume.sh
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
> 2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,926 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VhdProcessor - VHD Processor
> 2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.IsoProcessor - ISO Processor
> 2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
> 2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VmdkProcessor - VMDK Processor
> 2013-01-24 13:51:44,929 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.RawImageProcessor - Raw Image Processor
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
> 2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
> 2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
> 2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
> java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
> at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
> at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
> at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
> at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-01-24 13:51:44,954 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) server resources successfully discovered by SecondaryStorage
> 2013-01-24 13:51:44,954 WARN [cloud.api.ApiDispatcher] (catalina-exec-10:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
> 2013-01-24 13:51:55,168 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
>
>
> Best regards,
>
> Ben Linton
> Network Operations
> Hivelocity Ventures Corp.
> http://www.hivelocity.net
> 1-888-869(HOST)4678 x 222
> Fax (813) 902-6477
> Skype/AIM - HivelocityBen
> MSN - ben@hivelocity.net
>
>
RE: Issue with secondary storage on Cloudstack 4 - Host sent
incorrect data center: null
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
I had exactly the same problem, and the only thing saved me in my test environment the fact I was running the management and database servers on virtual so I had a snapshot to revert to.
After upgrade from 3.0.2 to 4.0.0 my CS jumped off the cliff, systemVM templates self-destructed, could not remove/add secondary storage because of the same error: "Host sent incorrect data center: null"
The only thing I can suggest is to ditch your current install, build a 3.0.2 and disable storage clean-up in global config as it can result in killing your snapshots and remove actually required templates.
Apart from that 3.0.2 seems solid, in production with it for about a year now, just works, everyone is happy.
I've have also hacked the 4.0 console-proxy back into the 3.0.2 systemISO so it follows resolution changes in 3.0.2 too.
After going into production make sure you have a test environment for testing upgrades and new features.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk
Follow us on Twitter: www.twitter.com/veberhost
Follow us on Facebook: www.facebook.com/veberhost
-----Original Message-----
From: Ben Linton [mailto:ben@hivelocity.net]
Sent: 24 January 2013 19:21
To: cloudstack-users@incubator.apache.org
Subject: Issue with secondary storage on Cloudstack 4 - Host sent incorrect data center: null
Hello,
I am having a strange problem adding secondary storage to our cloudstack system. I've had the same secondary storage added in the past and I only recently removed while doing some clean up and repair. However now we cannot add it back. I am using the management server for secondary storage and I have cleared out the directory and re-downloaded the xen system vm template image. I am using the private IP address of the management server and /export/secondary when trying to add secondary storage
Once I try to add it fails immediately, I will post the full log at the bottom but these are some entries that could pertain to why it is failing --
Once I hit "okay these are the following entries -
2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
Then it searches for a bunch of scripts and we arrive at this point and fails totally
cessor
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
For a little background -
Our cloudstack system consists of one management server and 2 xen hypervisors setup using advanced networking and multiple primary storage targets. This was a working system on 3.0.2(and prior versions).
Our latest issue stemmed from an issue with a failed primary storage iscsi target. Our system VMs were trying to boot using this target and we had quite a time removing this old failed target. During this process we upgraded to 4.x to try to use the "Force" option on primary storage. I eventually removed the primary storage entry from the database using a procedure I found online. After that the system VMs weren't trying to boot off of that target however there was some issues with secondary storage so I tried to remove and re-add as this fixed a similar issue we had in the past. However now we can't add storage back and I've search and tried many different things to no avail.
I am about ready to dump the entire system and reload. I would rather like to get this issue resolved as we are going to build a production cloudstack and it would be nice to see we don't encounter issues that we can't resolve.
If there are other details needed let me know. Any help or advice is certainly appreciated.
2013-01-24 13:51:44,829 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Trying to add a new host at nfs://10.0.0.15/export/secondary in data center 1
2013-01-24 13:51:44,830 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo mount 10.0.0.15:/export/secondary /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,870 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,871 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Executing: sudo umount /var/lib/cloud/mnt/67b2f964
2013-01-24 13:51:44,902 DEBUG [storage.secondary.SecondaryStorageDiscoverer] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, what's going on? eth1ip is null!!
2013-01-24 13:51:44,912 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Wait, there is no storageip in /proc/cmdline, something wrong!
2013-01-24 13:51:44,912 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr0
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Unable to get network interface for xenbr1
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Retrieving network interface: null
2013-01-24 13:51:44,913 ERROR [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Nics are not configured!
2013-01-24 13:51:44,913 INFO [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Designating private to be nic eth1
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in the classpath
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/storage/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./scripts/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/./config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/java/config_ssl.sh
2013-01-24 13:51:44,914 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/share/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/storage/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./scripts/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/./config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/common/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/cloud/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/lib64/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /usr/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/storage/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././scripts/config_ssl.sh
2013-01-24 13:51:44,915 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/././config_ssl.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_ssl.sh in /var/lib/cloud/management/./config_ssl.sh
2013-01-24 13:51:44,916 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_ssl.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in the classpath
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/storage/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./scripts/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/./config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/java/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/share/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
2013-01-24 13:51:44,916 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/storage/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./scripts/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/./config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/common/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/cloud/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/lib64/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /usr/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/secondary/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/storage/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././scripts/config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/././config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for config_auth.sh in /var/lib/cloud/management/./config_auth.sh
2013-01-24 13:51:44,917 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script config_auth.sh
2013-01-24 13:51:44,917 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in the classpath
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./scripts/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/./ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/java/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/share/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./scripts/ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/./ipfirewall.sh
2013-01-24 13:51:44,918 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/common/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/cloud/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/lib64/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /usr/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in the current directory
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/secondary/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/storage/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././scripts/ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/././ipfirewall.sh
2013-01-24 13:51:44,919 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for ipfirewall.sh in /var/lib/cloud/management/./ipfirewall.sh
2013-01-24 13:51:44,919 WARN [utils.script.Script] (catalina-exec-10:null) Unable to find script ipfirewall.sh
2013-01-24 13:51:44,919 INFO [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Secondary storage runs in role templateProcessor
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in the classpath
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,922 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/storage/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/scripts/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/java/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/share/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvmtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,923 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvmtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvmtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in the classpath
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,923 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/storage/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/scripts/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/java/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/share/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createtmplt.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createtmplt.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createtmplt.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in the classpath
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,924 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/storage/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/scripts/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/java/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/share/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for listvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,925 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) listvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/listvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in the classpath
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) System resource: null
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Current binaries reside at /usr/share/java
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/storage/createvolume.sh
2013-01-24 13:51:44,925 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/scripts/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/java/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/share/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /createvolume.sh
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Searching in environment.properties
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) environment.properties says scripts should be in /usr/lib64/cloud/common
2013-01-24 13:51:44,926 DEBUG [utils.script.Script] (catalina-exec-10:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,926 INFO [storage.template.DownloadManagerImpl] (catalina-exec-10:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VhdProcessor - VHD Processor
2013-01-24 13:51:44,927 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.IsoProcessor - ISO Processor
2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.QCOW2Processor - QCOW2 Processor
2013-01-24 13:51:44,928 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.VmdkProcessor - VMDK Processor
2013-01-24 13:51:44,929 INFO [utils.component.ComponentLocator] (catalina-exec-10:null) Found component: com.cloud.storage.template.Processor in com.cloud.storage.template.RawImageProcessor - Raw Image Processor
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for private nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [cloud.resource.ServerResourceBase] (catalina-exec-10:null) Parameters for storage nic: 10.0.0.15 - b8:ac:6f:8f:2f:6d-255.255.255.0
2013-01-24 13:51:44,933 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2013-01-24 13:51:44,943 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-10:null) Execution is successful.
2013-01-24 13:51:44,945 DEBUG [agent.transport.Request] (catalina-exec-10:null) Seq -1--1: Startup request from directly connected host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 111, [{"type":"SecondaryStorage","iqn":"NoIqn","privateIpAddress":"10.0.0.15","privateMacAddress":"b8:ac:6f:8f:2f:6d","privateNetmask":"255.255.255.0","storageIpAddress":"10.0.0.15","storageNetmask":"255.255.255.0","storageMacAddress":"b8:ac:6f:8f:2f:6d","wait":0}] }
2013-01-24 13:51:44,950 WARN [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) Unable to connect due to
java.lang.IllegalArgumentException: Host 10.0.0.15 sent incorrect data center: null
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1501)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1629)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:720)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:551)
at com.cloud.api.commands.AddSecondaryStorageCmd.execute(AddSecondaryStorageCmd.java:79)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-01-24 13:51:44,954 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-10:null) server resources successfully discovered by SecondaryStorage
2013-01-24 13:51:44,954 WARN [cloud.api.ApiDispatcher] (catalina-exec-10:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
2013-01-24 13:51:55,168 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
Best regards,
Ben Linton
Network Operations
Hivelocity Ventures Corp.
http://www.hivelocity.net
1-888-869(HOST)4678 x 222
Fax (813) 902-6477
Skype/AIM - HivelocityBen
MSN - ben@hivelocity.net