You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users-cn@cloudstack.apache.org by "linuxbqj@gmail.com" <li...@gmail.com> on 2014/07/14 09:41:19 UTC
Builds and runs a CloudStack Management Server stack under Docker.
https://github.com/fifthecho/docker-cloudstack
Apache CloudStack in Docker
HOWTO:
git clone the file to your computer
git clone https://github.com/fifthecho/docker-cloudstack.git
Set up Ansible [http://docs.ansible.com/intro_installation.html]
Execute ./build-and-run.sh
The first Playbook (build-docker-container.yml) is executed to create
the Dockerfile and builds the container. This playbook also ensures
that NFS is installed and creates /opt/storage/primary,
/opt/storage/secondary adds them to /etc/exports, restarts the NFS
server. Once all that is done, the playbook starts the container with
Secondary Storage mounted.
The second Playbook (configure-environment.yml) is executed to set
everything up on your host and create an Ansible host inventory for
the Docker container
The third Playbook (setup-cloudstack.yml) does the "heavy lifting" of
getting ACS management up and running. This includes setting up the
ACS database, starting ACS, and installing the system vm templates for
KVM and XenServer.
Access the CloudStack management interface via http://localhost:8080/client
Cloud!
Additonal work is forthcoming to provide a HTTP server to proxy
traffic to prevent the ugly jetty errors you get if you hit
http://localhost:8080/
The supervisord web interface is exposed on port 9001 from the
container (local port can be fetched by running "docker port
cloudstack 9001") for viewing CloudStack logs. I may move to Circus
from Supervisord to provide the ACS logs in the "docker logs" output.
Username and password are both "cloudstack"
Additionally, this is raw, OSS Apache CloudStack built from source and
not a noredist build (like the RPMs/DEBs), so it won't work with
things like VMware, Netscaler, etc., but I may fix that too.
Pull requests are very welcome.
--
白清杰 (Born Bai)
Mail: linuxbqj@gmail.com
敬畏耶和华是智慧的开端
回复： 回复：RE: 注册模板后出现状态：Storage agent or storage VM disconnected
Posted by 赵涛 <zh...@qq.com>.
在ssvm里查看/var/log/cloud/systemvm.log日志 如下：
2014-07-16 06:17:43,730 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: dest ip 10.11.16.0/24 is in the same subnet as eth1 ip 10.11.16.51
2014-07-16 06:17:43,730 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.17.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:43,734 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,735 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.17.0/24
2014-07-16 06:17:43,735 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.17.0/24
2014-07-16 06:17:43,737 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,737 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.17.0/24 via 10.11.16.254
2014-07-16 06:17:43,739 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,739 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.17.0/24 via 10.11.16.254
2014-07-16 06:17:43,739 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.24.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:43,743 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,743 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.24.0/24
2014-07-16 06:17:43,743 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.24.0/24
2014-07-16 06:17:43,745 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,745 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.24.0/24 via 10.11.16.254
2014-07-16 06:17:43,748 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,748 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.24.0/24 via 10.11.16.254
2014-07-16 06:17:43,748 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 192.168.1.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:43,752 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,752 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=192.168.1.0/24
2014-07-16 06:17:43,752 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 192.168.1.0/24
2014-07-16 06:17:43,754 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,754 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 192.168.1.0/24 via 10.11.16.254
2014-07-16 06:17:43,756 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,756 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=192.168.1.0/24 via 10.11.16.254
2014-07-16 06:17:43,756 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: /usr/local/cloud/systemvm/config_auth.sh cloud nD7iafqpzcsxqme
2014-07-16 06:17:43,762 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:43,762 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Adding password for user cloud
2014-07-16 06:17:43,763 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 46-101646339: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-16 06:17:43,832 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Request:Seq 46-101646340: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:17:43,832 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.storage.ListTemplateCommand
2014-07-16 06:17:43,832 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: mount
2014-07-16 06:17:43,834 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:17:43,836 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/
2014-07-16 06:17:44,154 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:17:44,154 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) found 28 templates[/mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261/c90f1ba1-8610-3dbb-bb6e-2bcad6ff75be.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268/c036ba8f-7976-342b-a82a-2fb58d5724ac.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso]
2014-07-16 06:17:44,156 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: centos55-x86_64, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2
2014-07-16 06:17:44,158 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: routing-3, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2
2014-07-16 06:17:44,160 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2
2014-07-16 06:17:44,162 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254
2014-07-16 06:17:44,169 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209
2014-07-16 06:17:44,175 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216
2014-07-16 06:17:44,182 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264
2014-07-16 06:17:44,188 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261
2014-07-16 06:17:44,194 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 217-2-0245468c-5e25-3513-92a4-89b4a045751f, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2
2014-07-16 06:17:44,196 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287
2014-07-16 06:17:44,202 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280
2014-07-16 06:17:44,208 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273
2014-07-16 06:17:44,214 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249
2014-07-16 06:17:44,220 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258
2014-07-16 06:17:44,226 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263
2014-07-16 06:17:44,231 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283
2014-07-16 06:17:44,237 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267
2014-07-16 06:17:44,243 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282
2014-07-16 06:17:44,248 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208
2014-07-16 06:17:44,254 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265
2014-07-16 06:17:44,260 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215
2014-07-16 06:17:44,266 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210
2014-07-16 06:17:44,272 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211
2014-07-16 06:17:44,277 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262
2014-07-16 06:17:44,284 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268
2014-07-16 06:17:44,290 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252
2014-07-16 06:17:44,296 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212
2014-07-16 06:17:44,303 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 221-2-3706787c-d305-39e0-9680-9f84359df07a, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso
2014-07-16 06:17:44,304 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Seq 46-101646340: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:17:44,676 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Request:Seq 46-101646341: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:17:44,676 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Processing command: com.cloud.agent.api.storage.ListVolumeCommand
2014-07-16 06:17:44,677 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Executing: mount
2014-07-16 06:17:44,680 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:17:44,681 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvolume.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/volumes/
2014-07-16 06:17:44,691 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:17:44,691 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) found 0 volumes[]
2014-07-16 06:17:44,691 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Seq 46-101646341: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:17:44,709 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Location 1: Socket Socket[addr=/10.11.24.4,port=8250,localport=50806] closed on read. Probably -1 returned: Connection closed with -1 on reading size.
2014-07-16 06:17:44,710 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Closing socket Socket[addr=/10.11.24.4,port=8250,localport=50806]
2014-07-16 06:17:44,710 DEBUG [cloud.agent.Agent] (Agent-Handler-4:) Clearing watch list: 1
2014-07-16 06:17:46,179 INFO [cloud.agent.Agent] (Agent-Handler-5:) Connected to the server
2014-07-16 06:17:49,710 INFO [cloud.agent.Agent] (Agent-Handler-4:) Lost connection to the server. Dealing with the remaining commands...
2014-07-16 06:17:54,712 INFO [cloud.agent.Agent] (Agent-Handler-4:) Reconnecting...
2014-07-16 06:17:54,712 INFO [utils.nio.NioClient] (Agent-Selector:) Connecting to 10.11.24.4:8250
2014-07-16 06:17:54,802 INFO [utils.nio.NioClient] (Agent-Selector:) SSL: Handshake done
2014-07-16 06:17:54,803 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for private nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:17:54,803 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for storage nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:17:54,803 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for pubic nic: 10.11.17.32 - 06:c4:0c:00:00:48-255.255.255.0
2014-07-16 06:17:54,803 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2014-07-16 06:17:54,806 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:17:54,806 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Executing: hostname
2014-07-16 06:17:54,807 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:17:54,808 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq 46-328: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-914-VM-NfsSecondaryStorageResource","name":"s-914-VM","id":46,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.32","publicNetmask":"255.255.255.0","publicMacAddress":"06:c4:0c:00:00:48","privateIpAddress":"10.11.16.51","privateMacAddress":"06:5e:88:00:00:01","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.51","storageNetmask":"255.255.255.0","storageMacAddress":"06:5e:88:00:00:01","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-16 06:17:54,808 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Startup task created
2014-07-16 06:17:55,007 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Received response: Seq 46-328: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100100, [{"StartupAnswer":{"hostId":46,"pingInterval":60,"result":true,"wait":0}}] }
2014-07-16 06:17:55,007 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Startup task cancelled
2014-07-16 06:17:55,008 INFO [cloud.agent.Agent] (Agent-Handler-2:) Proccess agent startup answer, agent id = 46
2014-07-16 06:17:55,008 INFO [cloud.agent.Agent] (Agent-Handler-2:) Set agent id 46
2014-07-16 06:17:55,008 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Adding a watch list
2014-07-16 06:17:55,008 INFO [cloud.agent.Agent] (Agent-Handler-2:) Startup Response Received: agent id = 46
2014-07-16 06:17:55,008 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping: Seq 46-329: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 11, [{"PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":46,"wait":0}}] }
2014-07-16 06:17:55,019 DEBUG [cloud.agent.Agent] (Agent-Handler-3:) Received response: Seq 46-329: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100010, [{"PingAnswer":{"_command":{"hostType":"Storage","hostId":46,"wait":0},"result":true,"wait":0}}] }
2014-07-16 06:17:55,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Request:Seq 46-1303117825: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","certs":{},"wait":0}}] }
2014-07-16 06:17:55,021 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Processing command: com.cloud.agent.api.SecStorageSetupCommand
2014-07-16 06:17:55,021 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.80, eth1mask=255.255.255.0,destIp=10.11.18.7
2014-07-16 06:17:55,021 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route delete 10.11.18.7
2014-07-16 06:17:55,023 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:17:55,023 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route add 10.11.18.7 via 10.11.16.254
2014-07-16 06:17:55,025 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:17:55,025 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: added route to internal ip=10.11.18.7 via 10.11.16.254
2014-07-16 06:17:55,026 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: mount
2014-07-16 06:17:55,027 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:17:55,027 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Executing: /usr/local/cloud/systemvm/config_ssl.sh -i 10.11.17.32 -h s-914-VM -k /tmp/prvkey7579279569296277166.tmp -p /tmp/pubcert3403794863871149570.tmp
2014-07-16 06:17:57,188 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:17:57,188 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Stopping web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
... waiting ..
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
.
2014-07-16 06:17:57,188 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Seq 46-1303117825: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-16 06:17:57,290 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 46-1303117826: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"isAppendAIp":false,"wait":0}}] }
2014-07-16 06:17:57,290 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:17:57,290 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh false
2014-07-16 06:17:57,294 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:17:57,294 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 46-1303117826: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:17:57,364 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Request:Seq 46-1303117827: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-16 06:17:57,364 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Processing command: com.cloud.agent.api.SecStorageVMSetupCommand
2014-07-16 06:17:57,364 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.18.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,369 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,369 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.18.0/24
2014-07-16 06:17:57,369 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.18.0/24
2014-07-16 06:17:57,371 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,371 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.18.0/24 via 10.11.16.254
2014-07-16 06:17:57,373 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,374 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.18.0/24 via 10.11.16.254
2014-07-16 06:17:57,374 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.19.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,378 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,378 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.19.0/24
2014-07-16 06:17:57,378 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.19.0/24
2014-07-16 06:17:57,380 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,380 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.19.0/24 via 10.11.16.254
2014-07-16 06:17:57,382 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,382 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.19.0/24 via 10.11.16.254
2014-07-16 06:17:57,382 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.16.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,386 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,386 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.16.0/24
2014-07-16 06:17:57,386 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: dest ip 10.11.16.0/24 is in the same subnet as eth1 ip 10.11.16.51
2014-07-16 06:17:57,386 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.17.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,390 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,390 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.17.0/24
2014-07-16 06:17:57,390 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.17.0/24
2014-07-16 06:17:57,392 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,393 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.17.0/24 via 10.11.16.254
2014-07-16 06:17:57,395 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,395 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.17.0/24 via 10.11.16.254
2014-07-16 06:17:57,395 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.24.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,399 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,399 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.24.0/24
2014-07-16 06:17:57,399 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.24.0/24
2014-07-16 06:17:57,401 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,401 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.24.0/24 via 10.11.16.254
2014-07-16 06:17:57,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.24.0/24 via 10.11.16.254
2014-07-16 06:17:57,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 192.168.1.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:17:57,407 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,407 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=192.168.1.0/24
2014-07-16 06:17:57,407 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 192.168.1.0/24
2014-07-16 06:17:57,409 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,409 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 192.168.1.0/24 via 10.11.16.254
2014-07-16 06:17:57,411 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,412 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=192.168.1.0/24 via 10.11.16.254
2014-07-16 06:17:57,412 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: /usr/local/cloud/systemvm/config_auth.sh cloud nD7iafqpzcsxqme
2014-07-16 06:17:57,417 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:17:57,417 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Adding password for user cloud
2014-07-16 06:17:57,417 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 46-1303117827: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-16 06:17:57,481 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Request:Seq 46-1303117828: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:17:57,481 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.storage.ListTemplateCommand
2014-07-16 06:17:57,482 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: mount
2014-07-16 06:17:57,484 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:17:57,485 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/
2014-07-16 06:17:57,765 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:17:57,765 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) found 28 templates[/mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261/c90f1ba1-8610-3dbb-bb6e-2bcad6ff75be.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268/c036ba8f-7976-342b-a82a-2fb58d5724ac.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso]
2014-07-16 06:17:57,767 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: centos55-x86_64, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2
2014-07-16 06:17:57,770 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: routing-3, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2
2014-07-16 06:17:57,772 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2
2014-07-16 06:17:57,774 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254
2014-07-16 06:17:57,780 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209
2014-07-16 06:17:57,785 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216
2014-07-16 06:17:57,790 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264
2014-07-16 06:17:57,796 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261
2014-07-16 06:17:57,803 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 217-2-0245468c-5e25-3513-92a4-89b4a045751f, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2
2014-07-16 06:17:57,805 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287
2014-07-16 06:17:57,811 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280
2014-07-16 06:17:57,816 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273
2014-07-16 06:17:57,821 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249
2014-07-16 06:17:57,827 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258
2014-07-16 06:17:57,832 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263
2014-07-16 06:17:57,838 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283
2014-07-16 06:17:57,843 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267
2014-07-16 06:17:57,849 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282
2014-07-16 06:17:57,855 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208
2014-07-16 06:17:57,861 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265
2014-07-16 06:17:57,867 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215
2014-07-16 06:17:57,872 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210
2014-07-16 06:17:57,878 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211
2014-07-16 06:17:57,883 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262
2014-07-16 06:17:57,889 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268
2014-07-16 06:17:57,894 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252
2014-07-16 06:17:57,899 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212
2014-07-16 06:17:57,905 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 221-2-3706787c-d305-39e0-9680-9f84359df07a, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso
2014-07-16 06:17:57,906 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Seq 46-1303117828: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:17:58,225 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Request:Seq 46-1303117829: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:17:58,226 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Processing command: com.cloud.agent.api.storage.ListVolumeCommand
2014-07-16 06:17:58,226 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Executing: mount
2014-07-16 06:17:58,228 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:17:58,229 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvolume.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/volumes/
2014-07-16 06:17:58,238 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:17:58,238 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) found 0 volumes[]
2014-07-16 06:17:58,238 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Seq 46-1303117829: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:17:58,269 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Location 1: Socket Socket[addr=/10.11.24.4,port=8250,localport=50807] closed on read. Probably -1 returned: Connection closed with -1 on reading size.
2014-07-16 06:17:58,269 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Closing socket Socket[addr=/10.11.24.4,port=8250,localport=50807]
2014-07-16 06:17:58,269 DEBUG [cloud.agent.Agent] (Agent-Handler-4:) Clearing watch list: 1
2014-07-16 06:17:59,803 INFO [cloud.agent.Agent] (Agent-Handler-4:) Connected to the server
2014-07-16 06:18:03,270 INFO [cloud.agent.Agent] (Agent-Handler-4:) Lost connection to the server. Dealing with the remaining commands...
2014-07-16 06:18:08,271 INFO [cloud.agent.Agent] (Agent-Handler-4:) Reconnecting...
2014-07-16 06:18:08,271 INFO [utils.nio.NioClient] (Agent-Selector:) Connecting to 10.11.24.4:8250
2014-07-16 06:18:08,362 INFO [utils.nio.NioClient] (Agent-Selector:) SSL: Handshake done
2014-07-16 06:18:08,363 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for private nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:08,363 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for storage nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:08,363 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for pubic nic: 10.11.17.32 - 06:c4:0c:00:00:48-255.255.255.0
2014-07-16 06:18:08,363 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2014-07-16 06:18:08,365 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:08,366 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Executing: hostname
2014-07-16 06:18:08,367 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:08,367 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq 46-330: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-914-VM-NfsSecondaryStorageResource","name":"s-914-VM","id":46,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.32","publicNetmask":"255.255.255.0","publicMacAddress":"06:c4:0c:00:00:48","privateIpAddress":"10.11.16.51","privateMacAddress":"06:5e:88:00:00:01","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.51","storageNetmask":"255.255.255.0","storageMacAddress":"06:5e:88:00:00:01","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-16 06:18:08,367 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Startup task created
2014-07-16 06:18:08,567 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Received response: Seq 46-330: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100100, [{"StartupAnswer":{"hostId":46,"pingInterval":60,"result":true,"wait":0}}] }
2014-07-16 06:18:08,568 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Startup task cancelled
2014-07-16 06:18:08,568 INFO [cloud.agent.Agent] (Agent-Handler-2:) Proccess agent startup answer, agent id = 46
2014-07-16 06:18:08,568 INFO [cloud.agent.Agent] (Agent-Handler-2:) Set agent id 46
2014-07-16 06:18:08,568 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Adding a watch list
2014-07-16 06:18:08,568 INFO [cloud.agent.Agent] (Agent-Handler-2:) Startup Response Received: agent id = 46
2014-07-16 06:18:08,568 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping: Seq 46-331: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 11, [{"PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":46,"wait":0}}] }
2014-07-16 06:18:08,571 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Request:Seq 46-1683816449: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","certs":{},"wait":0}}] }
2014-07-16 06:18:08,571 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Processing command: com.cloud.agent.api.SecStorageSetupCommand
2014-07-16 06:18:08,571 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.80, eth1mask=255.255.255.0,destIp=10.11.18.7
2014-07-16 06:18:08,571 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route delete 10.11.18.7
2014-07-16 06:18:08,574 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:08,574 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route add 10.11.18.7 via 10.11.16.254
2014-07-16 06:18:08,576 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:08,576 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: added route to internal ip=10.11.18.7 via 10.11.16.254
2014-07-16 06:18:08,576 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: mount
2014-07-16 06:18:08,578 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:08,578 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Executing: /usr/local/cloud/systemvm/config_ssl.sh -i 10.11.17.32 -h s-914-VM -k /tmp/prvkey896557704488795261.tmp -p /tmp/pubcert6295364737312068298.tmp
2014-07-16 06:18:08,615 DEBUG [cloud.agent.Agent] (Agent-Handler-4:) Received response: Seq 46-331: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100010, [{"PingAnswer":{"_command":{"hostType":"Storage","hostId":46,"wait":0},"result":true,"wait":0}}] }
2014-07-16 06:18:09,780 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:09,781 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Stopping web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
... waiting .
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
.
2014-07-16 06:18:09,781 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Seq 46-1683816449: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:09,875 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 46-1683816450: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"isAppendAIp":false,"wait":0}}] }
2014-07-16 06:18:09,875 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:18:09,875 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh false
2014-07-16 06:18:09,879 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:18:09,879 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 46-1683816450: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:18:09,948 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Request:Seq 46-1683816451: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-16 06:18:09,948 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Processing command: com.cloud.agent.api.SecStorageVMSetupCommand
2014-07-16 06:18:09,948 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.18.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,952 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,952 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.18.0/24
2014-07-16 06:18:09,952 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.18.0/24
2014-07-16 06:18:09,955 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,955 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:09,957 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,957 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:09,957 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.19.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,961 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,961 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.19.0/24
2014-07-16 06:18:09,961 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.19.0/24
2014-07-16 06:18:09,963 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,963 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:09,965 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,965 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:09,965 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.16.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,969 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,970 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.16.0/24
2014-07-16 06:18:09,970 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: dest ip 10.11.16.0/24 is in the same subnet as eth1 ip 10.11.16.51
2014-07-16 06:18:09,970 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.17.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,974 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,974 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.17.0/24
2014-07-16 06:18:09,974 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.17.0/24
2014-07-16 06:18:09,976 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,977 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:09,979 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,979 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:09,979 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.24.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,983 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,983 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.24.0/24
2014-07-16 06:18:09,983 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 10.11.24.0/24
2014-07-16 06:18:09,985 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,985 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:09,987 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,987 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:09,987 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 192.168.1.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:09,991 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,991 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=192.168.1.0/24
2014-07-16 06:18:09,991 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route delete 192.168.1.0/24
2014-07-16 06:18:09,993 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,993 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: /bin/bash -c ip route add 192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:09,995 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:09,995 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) addRouteToInternalIp: added route to internal ip=192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:09,995 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: /usr/local/cloud/systemvm/config_auth.sh cloud nD7iafqpzcsxqme
2014-07-16 06:18:10,001 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:10,001 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Adding password for user cloud
2014-07-16 06:18:10,001 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 46-1683816451: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-16 06:18:10,066 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Request:Seq 46-1683816452: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:10,066 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.storage.ListTemplateCommand
2014-07-16 06:18:10,067 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: mount
2014-07-16 06:18:10,068 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:10,070 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/
2014-07-16 06:18:10,349 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:10,349 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) found 28 templates[/mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261/c90f1ba1-8610-3dbb-bb6e-2bcad6ff75be.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268/c036ba8f-7976-342b-a82a-2fb58d5724ac.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso]
2014-07-16 06:18:10,351 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: centos55-x86_64, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2
2014-07-16 06:18:10,353 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: routing-3, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2
2014-07-16 06:18:10,355 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2
2014-07-16 06:18:10,357 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254
2014-07-16 06:18:10,363 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209
2014-07-16 06:18:10,369 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216
2014-07-16 06:18:10,374 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264
2014-07-16 06:18:10,380 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261
2014-07-16 06:18:10,385 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 217-2-0245468c-5e25-3513-92a4-89b4a045751f, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2
2014-07-16 06:18:10,387 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287
2014-07-16 06:18:10,393 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280
2014-07-16 06:18:10,399 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273
2014-07-16 06:18:10,404 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249
2014-07-16 06:18:10,410 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258
2014-07-16 06:18:10,416 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263
2014-07-16 06:18:10,421 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283
2014-07-16 06:18:10,427 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267
2014-07-16 06:18:10,432 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282
2014-07-16 06:18:10,438 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208
2014-07-16 06:18:10,443 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265
2014-07-16 06:18:10,449 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215
2014-07-16 06:18:10,455 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210
2014-07-16 06:18:10,460 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211
2014-07-16 06:18:10,466 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262
2014-07-16 06:18:10,471 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268
2014-07-16 06:18:10,477 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252
2014-07-16 06:18:10,483 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212
2014-07-16 06:18:10,488 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-3:) Added template name: 221-2-3706787c-d305-39e0-9680-9f84359df07a, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso
2014-07-16 06:18:10,489 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Seq 46-1683816452: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:10,823 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Request:Seq 46-1683816453: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:10,823 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Processing command: com.cloud.agent.api.storage.ListVolumeCommand
2014-07-16 06:18:10,824 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Executing: mount
2014-07-16 06:18:10,826 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:10,827 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvolume.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/volumes/
2014-07-16 06:18:10,836 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:10,836 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) found 0 volumes[]
2014-07-16 06:18:10,836 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Seq 46-1683816453: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:10,855 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Location 1: Socket Socket[addr=/10.11.24.4,port=8250,localport=50808] closed on read. Probably -1 returned: Connection closed with -1 on reading size.
2014-07-16 06:18:10,856 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Closing socket Socket[addr=/10.11.24.4,port=8250,localport=50808]
2014-07-16 06:18:10,856 DEBUG [cloud.agent.Agent] (Agent-Handler-4:) Clearing watch list: 1
2014-07-16 06:18:13,363 INFO [cloud.agent.Agent] (Agent-Handler-4:) Connected to the server
2014-07-16 06:18:15,856 INFO [cloud.agent.Agent] (Agent-Handler-4:) Lost connection to the server. Dealing with the remaining commands...
2014-07-16 06:18:20,858 INFO [cloud.agent.Agent] (Agent-Handler-4:) Reconnecting...
2014-07-16 06:18:20,858 INFO [utils.nio.NioClient] (Agent-Selector:) Connecting to 10.11.24.4:8250
2014-07-16 06:18:20,946 INFO [utils.nio.NioClient] (Agent-Selector:) SSL: Handshake done
2014-07-16 06:18:20,947 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for private nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:20,947 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for storage nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:20,948 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for pubic nic: 10.11.17.32 - 06:c4:0c:00:00:48-255.255.255.0
2014-07-16 06:18:20,948 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2014-07-16 06:18:20,950 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:20,950 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Executing: hostname
2014-07-16 06:18:20,951 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:20,952 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq 46-332: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-914-VM-NfsSecondaryStorageResource","name":"s-914-VM","id":46,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.32","publicNetmask":"255.255.255.0","publicMacAddress":"06:c4:0c:00:00:48","privateIpAddress":"10.11.16.51","privateMacAddress":"06:5e:88:00:00:01","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.51","storageNetmask":"255.255.255.0","storageMacAddress":"06:5e:88:00:00:01","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-16 06:18:20,952 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Startup task created
2014-07-16 06:18:21,167 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Received response: Seq 46-332: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100100, [{"StartupAnswer":{"hostId":46,"pingInterval":60,"result":true,"wait":0}}] }
2014-07-16 06:18:21,167 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Startup task cancelled
2014-07-16 06:18:21,167 INFO [cloud.agent.Agent] (Agent-Handler-2:) Proccess agent startup answer, agent id = 46
2014-07-16 06:18:21,167 INFO [cloud.agent.Agent] (Agent-Handler-2:) Set agent id 46
2014-07-16 06:18:21,167 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Adding a watch list
2014-07-16 06:18:21,168 INFO [cloud.agent.Agent] (Agent-Handler-2:) Startup Response Received: agent id = 46
2014-07-16 06:18:21,168 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping: Seq 46-333: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 11, [{"PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":46,"wait":0}}] }
2014-07-16 06:18:21,184 DEBUG [cloud.agent.Agent] (Agent-Handler-3:) Received response: Seq 46-333: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100010, [{"PingAnswer":{"_command":{"hostType":"Storage","hostId":46,"wait":0},"result":true,"wait":0}}] }
2014-07-16 06:18:21,188 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Request:Seq 46-1933115393: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","certs":{},"wait":0}}] }
2014-07-16 06:18:21,188 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Processing command: com.cloud.agent.api.SecStorageSetupCommand
2014-07-16 06:18:21,188 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.80, eth1mask=255.255.255.0,destIp=10.11.18.7
2014-07-16 06:18:21,188 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route delete 10.11.18.7
2014-07-16 06:18:21,191 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:21,191 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route add 10.11.18.7 via 10.11.16.254
2014-07-16 06:18:21,193 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:21,193 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: added route to internal ip=10.11.18.7 via 10.11.16.254
2014-07-16 06:18:21,193 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: mount
2014-07-16 06:18:21,196 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:21,197 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Executing: /usr/local/cloud/systemvm/config_ssl.sh -i 10.11.17.32 -h s-914-VM -k /tmp/prvkey2711851077634269814.tmp -p /tmp/pubcert2514594449977992293.tmp
2014-07-16 06:18:21,905 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 46-1933115394: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.3","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-16 06:18:21,905 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:18:21,905 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh true 10.11.19.3
2014-07-16 06:18:21,908 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:18:21,908 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 46-1933115394: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:18:22,336 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:22,336 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Stopping web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
... waiting .
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
.
2014-07-16 06:18:22,336 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Seq 46-1933115393: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:22,491 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Request:Seq 46-1933115395: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.3","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-16 06:18:22,491 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:18:22,491 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh false 10.11.19.3
2014-07-16 06:18:22,497 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:22,497 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 46-1933115395: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:18:22,587 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Request:Seq 46-1933115396: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-16 06:18:22,587 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.SecStorageVMSetupCommand
2014-07-16 06:18:22,587 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.18.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,592 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,592 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.18.0/24
2014-07-16 06:18:22,592 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.18.0/24
2014-07-16 06:18:22,594 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,594 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:22,596 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,596 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:22,596 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.19.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,600 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,601 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.19.0/24
2014-07-16 06:18:22,601 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.19.0/24
2014-07-16 06:18:22,603 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,603 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:22,605 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,605 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:22,605 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.16.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,609 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,609 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.16.0/24
2014-07-16 06:18:22,609 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: dest ip 10.11.16.0/24 is in the same subnet as eth1 ip 10.11.16.51
2014-07-16 06:18:22,609 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.17.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,613 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,613 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.17.0/24
2014-07-16 06:18:22,613 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.17.0/24
2014-07-16 06:18:22,616 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,616 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:22,618 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,618 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:22,618 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.24.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,622 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,622 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.24.0/24
2014-07-16 06:18:22,622 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.24.0/24
2014-07-16 06:18:22,624 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,624 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:22,626 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,626 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:22,626 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 192.168.1.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:22,630 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,630 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=192.168.1.0/24
2014-07-16 06:18:22,631 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 192.168.1.0/24
2014-07-16 06:18:22,633 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,633 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:22,635 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,635 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:22,635 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Executing: /usr/local/cloud/systemvm/config_auth.sh cloud nD7iafqpzcsxqme
2014-07-16 06:18:22,640 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:22,640 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Adding password for user cloud
2014-07-16 06:18:22,641 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Seq 46-1933115396: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-16 06:18:22,685 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Request:Seq 46-1933115397: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:22,686 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Processing command: com.cloud.agent.api.storage.ListTemplateCommand
2014-07-16 06:18:22,686 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Executing: mount
2014-07-16 06:18:22,688 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:22,689 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/
2014-07-16 06:18:22,969 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:22,969 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) found 28 templates[/mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261/c90f1ba1-8610-3dbb-bb6e-2bcad6ff75be.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268/c036ba8f-7976-342b-a82a-2fb58d5724ac.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso]
2014-07-16 06:18:22,971 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: centos55-x86_64, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2
2014-07-16 06:18:22,974 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: routing-3, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2
2014-07-16 06:18:22,976 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2
2014-07-16 06:18:22,978 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254
2014-07-16 06:18:22,984 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209
2014-07-16 06:18:22,990 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216
2014-07-16 06:18:22,995 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264
2014-07-16 06:18:23,001 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261
2014-07-16 06:18:23,006 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 217-2-0245468c-5e25-3513-92a4-89b4a045751f, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2
2014-07-16 06:18:23,008 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287
2014-07-16 06:18:23,014 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280
2014-07-16 06:18:23,019 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273
2014-07-16 06:18:23,025 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249
2014-07-16 06:18:23,030 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258
2014-07-16 06:18:23,035 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263
2014-07-16 06:18:23,041 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283
2014-07-16 06:18:23,047 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267
2014-07-16 06:18:23,052 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282
2014-07-16 06:18:23,058 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208
2014-07-16 06:18:23,063 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265
2014-07-16 06:18:23,068 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215
2014-07-16 06:18:23,074 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210
2014-07-16 06:18:23,079 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211
2014-07-16 06:18:23,085 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262
2014-07-16 06:18:23,090 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268
2014-07-16 06:18:23,096 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252
2014-07-16 06:18:23,102 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212
2014-07-16 06:18:23,107 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 221-2-3706787c-d305-39e0-9680-9f84359df07a, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso
2014-07-16 06:18:23,108 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Seq 46-1933115397: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:23,430 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 46-1933115398: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:23,430 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.storage.ListVolumeCommand
2014-07-16 06:18:23,431 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:) Executing: mount
2014-07-16 06:18:23,434 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:18:23,435 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-4:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvolume.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/volumes/
2014-07-16 06:18:23,444 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:18:23,445 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-4:) found 0 volumes[]
2014-07-16 06:18:23,445 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 46-1933115398: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:23,465 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Location 1: Socket Socket[addr=/10.11.24.4,port=8250,localport=50809] closed on read. Probably -1 returned: Connection closed with -1 on reading size.
2014-07-16 06:18:23,466 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Closing socket Socket[addr=/10.11.24.4,port=8250,localport=50809]
2014-07-16 06:18:23,466 DEBUG [cloud.agent.Agent] (Agent-Handler-5:) Clearing watch list: 1
2014-07-16 06:18:25,947 INFO [cloud.agent.Agent] (Agent-Handler-4:) Connected to the server
2014-07-16 06:18:28,467 INFO [cloud.agent.Agent] (Agent-Handler-5:) Lost connection to the server. Dealing with the remaining commands...
2014-07-16 06:18:33,468 INFO [cloud.agent.Agent] (Agent-Handler-5:) Reconnecting...
2014-07-16 06:18:33,469 INFO [utils.nio.NioClient] (Agent-Selector:) Connecting to 10.11.24.4:8250
2014-07-16 06:18:33,555 INFO [utils.nio.NioClient] (Agent-Selector:) SSL: Handshake done
2014-07-16 06:18:33,556 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for private nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:33,557 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for storage nic: 10.11.16.51 - 06:5e:88:00:00:01-255.255.255.0
2014-07-16 06:18:33,557 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:) Parameters for pubic nic: 10.11.17.32 - 06:c4:0c:00:00:48-255.255.255.0
2014-07-16 06:18:33,557 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2014-07-16 06:18:33,560 DEBUG [storage.resource.NfsSecondaryStorageResource] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:33,560 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Executing: hostname
2014-07-16 06:18:33,561 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Execution is successful.
2014-07-16 06:18:33,561 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Sending Startup: Seq 46-334: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-914-VM-NfsSecondaryStorageResource","name":"s-914-VM","id":46,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.32","publicNetmask":"255.255.255.0","publicMacAddress":"06:c4:0c:00:00:48","privateIpAddress":"10.11.16.51","privateMacAddress":"06:5e:88:00:00:01","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.51","storageNetmask":"255.255.255.0","storageMacAddress":"06:5e:88:00:00:01","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-16 06:18:33,561 DEBUG [cloud.agent.Agent] (Agent-Handler-1:) Startup task created
2014-07-16 06:18:33,762 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Received response: Seq 46-334: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100100, [{"StartupAnswer":{"hostId":46,"pingInterval":60,"result":true,"wait":0}}] }
2014-07-16 06:18:33,765 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Startup task cancelled
2014-07-16 06:18:33,765 INFO [cloud.agent.Agent] (Agent-Handler-2:) Proccess agent startup answer, agent id = 46
2014-07-16 06:18:33,765 INFO [cloud.agent.Agent] (Agent-Handler-2:) Set agent id 46
2014-07-16 06:18:33,765 DEBUG [cloud.agent.Agent] (Agent-Handler-2:) Adding a watch list
2014-07-16 06:18:33,765 INFO [cloud.agent.Agent] (Agent-Handler-2:) Startup Response Received: agent id = 46
2014-07-16 06:18:33,765 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping: Seq 46-335: { Cmd , MgmtId: -1, via: 46, Ver: v1, Flags: 11, [{"PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":46,"wait":0}}] }
2014-07-16 06:18:33,766 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Request:Seq 46-1592393729: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","certs":{},"wait":0}}] }
2014-07-16 06:18:33,766 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Processing command: com.cloud.agent.api.SecStorageSetupCommand
2014-07-16 06:18:33,766 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.80, eth1mask=255.255.255.0,destIp=10.11.18.7
2014-07-16 06:18:33,766 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route delete 10.11.18.7
2014-07-16 06:18:33,769 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:33,769 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: /bin/bash -c ip route add 10.11.18.7 via 10.11.16.254
2014-07-16 06:18:33,772 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:33,772 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) addRouteToInternalIp: added route to internal ip=10.11.18.7 via 10.11.16.254
2014-07-16 06:18:33,773 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: mount
2014-07-16 06:18:33,775 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:33,775 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Executing: /usr/local/cloud/systemvm/config_ssl.sh -i 10.11.17.32 -h s-914-VM -k /tmp/prvkey8460466906045843370.tmp -p /tmp/pubcert3535456960738248048.tmp
2014-07-16 06:18:33,808 DEBUG [cloud.agent.Agent] (Agent-Handler-4:) Received response: Seq 46-335: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100010, [{"PingAnswer":{"_command":{"hostType":"Storage","hostId":46,"wait":0},"result":true,"wait":0}}] }
2014-07-16 06:18:34,967 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:34,968 DEBUG [utils.script.Script] (agentRequest-Handler-5:) Stopping web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
... waiting .
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 10.11.17.32 for ServerName
.
2014-07-16 06:18:34,968 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Seq 46-1592393729: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:35,153 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Request:Seq 46-1592393730: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.3","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-16 06:18:35,153 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:18:35,153 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh false 10.11.19.3
2014-07-16 06:18:35,157 DEBUG [utils.script.Script] (agentRequest-Handler-2:) Execution is successful.
2014-07-16 06:18:35,158 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 46-1592393730: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:18:35,229 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Request:Seq 46-1592393731: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-16 06:18:35,229 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.SecStorageVMSetupCommand
2014-07-16 06:18:35,229 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.18.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,233 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,233 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.18.0/24
2014-07-16 06:18:35,234 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.18.0/24
2014-07-16 06:18:35,236 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,236 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:35,238 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,238 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.18.0/24 via 10.11.16.254
2014-07-16 06:18:35,238 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.19.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,242 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.19.0/24
2014-07-16 06:18:35,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.19.0/24
2014-07-16 06:18:35,245 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,245 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:35,247 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,247 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.19.0/24 via 10.11.16.254
2014-07-16 06:18:35,247 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.16.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,251 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,251 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.16.0/24
2014-07-16 06:18:35,251 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: dest ip 10.11.16.0/24 is in the same subnet as eth1 ip 10.11.16.51
2014-07-16 06:18:35,252 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.17.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,256 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,256 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.17.0/24
2014-07-16 06:18:35,256 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.17.0/24
2014-07-16 06:18:35,258 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,258 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:35,260 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,260 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.17.0/24 via 10.11.16.254
2014-07-16 06:18:35,260 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 10.11.24.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,265 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,265 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=10.11.24.0/24
2014-07-16 06:18:35,265 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 10.11.24.0/24
2014-07-16 06:18:35,267 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,267 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:35,269 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=10.11.24.0/24 via 10.11.16.254
2014-07-16 06:18:35,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c iptables -I OUTPUT -o eth1 -d 192.168.1.0/24 -p tcp -m state --state NEW -m tcp -j ACCEPT
2014-07-16 06:18:35,274 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,274 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: localgw=10.11.16.254, eth1ip=10.11.16.51, eth1mask=255.255.255.0,destIp=192.168.1.0/24
2014-07-16 06:18:35,274 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route delete 192.168.1.0/24
2014-07-16 06:18:35,276 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,276 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Executing: /bin/bash -c ip route add 192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:35,278 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,278 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) addRouteToInternalIp: added route to internal ip=192.168.1.0/24 via 10.11.16.254
2014-07-16 06:18:35,278 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Executing: /usr/local/cloud/systemvm/config_auth.sh cloud nD7iafqpzcsxqme
2014-07-16 06:18:35,285 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Execution is successful.
2014-07-16 06:18:35,285 DEBUG [utils.script.Script] (agentRequest-Handler-3:) Adding password for user cloud
2014-07-16 06:18:35,285 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Seq 46-1592393731: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-16 06:18:35,349 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Request:Seq 46-1592393732: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:35,349 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Processing command: com.cloud.agent.api.storage.ListTemplateCommand
2014-07-16 06:18:35,350 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Executing: mount
2014-07-16 06:18:35,352 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:35,354 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/
2014-07-16 06:18:35,559 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Request:Seq 46-1592393733: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.3","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-16 06:18:35,559 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Processing command: com.cloud.agent.api.SecStorageFirewallCfgCommand
2014-07-16 06:18:35,559 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Executing: /usr/local/cloud/systemvm/ipfirewall.sh true 10.11.19.3
2014-07-16 06:18:35,562 DEBUG [utils.script.Script] (agentRequest-Handler-4:) Execution is successful.
2014-07-16 06:18:35,563 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq 46-1592393733: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-16 06:18:35,660 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Execution is successful.
2014-07-16 06:18:35,660 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) found 28 templates[/mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261/c90f1ba1-8610-3dbb-bb6e-2bcad6ff75be.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268/c036ba8f-7976-342b-a82a-2fb58d5724ac.qcow2, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212/, /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso]
2014-07-16 06:18:35,663 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: centos55-x86_64, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/4/e970d345-e30b-30c4-879c-88d63200a721.qcow2
2014-07-16 06:18:35,666 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: routing-3, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/1/3/6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2
2014-07-16 06:18:35,669 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/218/d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2
2014-07-16 06:18:35,671 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/254
2014-07-16 06:18:35,678 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/209
2014-07-16 06:18:35,684 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/216
2014-07-16 06:18:35,690 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/264
2014-07-16 06:18:35,696 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/261
2014-07-16 06:18:35,702 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 217-2-0245468c-5e25-3513-92a4-89b4a045751f, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/217/3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2
2014-07-16 06:18:35,704 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/287
2014-07-16 06:18:35,710 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/280
2014-07-16 06:18:35,716 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/273
2014-07-16 06:18:35,722 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/249
2014-07-16 06:18:35,728 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/258
2014-07-16 06:18:35,734 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/263
2014-07-16 06:18:35,741 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/283
2014-07-16 06:18:35,747 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/267
2014-07-16 06:18:35,753 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/282
2014-07-16 06:18:35,759 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/208
2014-07-16 06:18:35,765 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/265
2014-07-16 06:18:35,771 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/215
2014-07-16 06:18:35,777 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/210
2014-07-16 06:18:35,783 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/211
2014-07-16 06:18:35,789 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/262
2014-07-16 06:18:35,795 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/268
2014-07-16 06:18:35,801 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/252
2014-07-16 06:18:35,807 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Post download installation was not completed for /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/212
2014-07-16 06:18:35,813 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:) Added template name: 221-2-3706787c-d305-39e0-9680-9f84359df07a, path: /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/template/tmpl/2/221/221-2-3706787c-d305-39e0-9680-9f84359df07a.iso
2014-07-16 06:18:35,814 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:) Seq 46-1592393732: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:36,179 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Request:Seq 46-1592393734: { Cmd , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-16 06:18:36,179 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Processing command: com.cloud.agent.api.storage.ListVolumeCommand
2014-07-16 06:18:36,180 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Executing: mount
2014-07-16 06:18:36,181 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:36,182 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-5:) Executing: /usr/local/cloud/systemvm/scripts/storage/secondary/listvolume.sh -r /mnt/SecStorage/a35b9ab9-d061-3ed7-bbe7-82022eec78e4/volumes/
2014-07-16 06:18:36,192 DEBUG [storage.template.DownloadManagerImpl] (agentRequest-Handler-5:) Execution is successful.
2014-07-16 06:18:36,192 INFO [storage.template.DownloadManagerImpl] (agentRequest-Handler-5:) found 0 volumes[]
2014-07-16 06:18:36,192 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:) Seq 46-1592393734: { Ans: , MgmtId: 73548179740843, via: 46, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-16 06:18:36,215 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Location 1: Socket Socket[addr=/10.11.24.4,port=8250,localport=50810] closed on read. Probably -1 returned: Connection closed with -1 on reading size.
2014-07-16 06:18:36,216 DEBUG [utils.nio.NioConnection] (Agent-Selector:) Closing socket Socket[addr=/10.11.24.4,port=8250,localport=50810]
2014-07-16 06:18:36,216 DEBUG [cloud.agent.Agent] (Agent-Handler-5:) Clearing watch list: 1
------------------ 原始邮件 ------------------
发件人: "Wei ZHOU";<us...@gmail.com>;
发送时间: 2014年7月15日(星期二) 晚上6:40
收件人: "users-cn"<us...@cloudstack.apache.org>;
主题: Re: 回复：RE: 注册模板后出现状态：Storage agent or storage VM disconnected
你看一下ssvm里面的log吧
/var/log/cloud/目录
2014-07-15 10:34 GMT+02:00 赵涛 <zh...@qq.com>:
> 在vm_instance表里 s-909-VM/ s-889-VM是Running ，但在
> host表里显示状态是Alerting，销毁重建后还是一样
> ------------------ 原始邮件 ------------------
> 发件人: "YUAN, GUI -HBIS";<yu...@hbis.com.cn>;
> 发送时间: 2014年7月15日(星期二) 下午2:58
> 收件人: "users-cn"<us...@cloudstack.apache.org>;
>
> 主题: RE: 注册模板后出现状态：Storage agent or storage VM disconnected
>
>
>
> 1.SSVM没有启动，可能是对应的AgentHost连不上。确认有可用的HOST
> 如：s-909-VM/ s-889-VM
>
> 2.如果SSVM正常，模板添加时，提示SSVM- response time out
> 则需要确认NFS服务器的NFS服务状态及防火墙设置
>
>
>
> -----Original Message-----
> From: 赵涛 [mailto:zhaotao_cn@qq.com]
> Sent: Tuesday, July 15, 2014 12:46 PM
> To: users-cn
> Subject: 注册模板后出现状态：Storage agent or storage VM disconnected
>
> 注册模板后出现状态：Storage agent or storage VM disconnected,删除SSVM后问题依旧，请大神指导
>
> 删除SSVM虚拟机后重建的SSVM虚拟机上报日志：
>
>
> 2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> SecurityGroupListener
> 2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener]
> (AgentManager-Handler-7:null) Received a host startup notification
> 2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> StoragePoolMonitor
> 2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> SecondaryStorageListener
> 2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener]
> (AgentManager-Handler-7:null) Received a host startup notification
> com.cloud.agent.api.StartupSecondaryStorageCommand
> 2014-07-15 10:41:46,714 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}]
> }
> 2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-14:null) Ping from 41
> 2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
> (AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
> SecStorageSetupAnswer } }
> 2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
> 2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.SecStorageSetupCommand: success
> 2014-07-15 10:41:46,809 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed secondary storage
> nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
> 2014-07-15 10:41:46,831 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
> }
> 2014-07-15 10:41:46,880 DEBUG [agent.transport.Request]
> (AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:46,881 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:46,881 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed firewall rules into
> SSVM s-889-VM
> 2014-07-15 10:41:46,891 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
> }
> 2014-07-15 10:41:46,901 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:46,902 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:46,902 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed firewall rules into
> s-909-VM
> 2014-07-15 10:41:46,929 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","
> 10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","
> 192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
> }
> 2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
> (AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"Answer":{"result":true,"details":"","wait":0}}] }
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
> 2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.SecStorageVMSetupCommand:
> 2014-07-15 10:41:47,083 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed http auth into
> s-909-VM
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener:
> NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener:
> AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
> 2014-07-15 10:41:47,101 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"storage.ListTemplateCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","wait":0}}] }
> 2014-07-15 10:41:47,629 DEBUG [agent.transport.Request]
> (AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"storage.ListTemplateAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
> 2014-07-15 10:41:47,630 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer
> } }
> 2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.storage.ListTemplateCommand: success
> 2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found routing-3 already in the
> template host table
> 2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already
> in the template host table
> 2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync did not find
> centos56-x86_64-xen ready on server 11, will request download to
> start/resume shortly
> 2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready
> on server 11, will request download to start/resume shortly
> 2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host
> table
> 2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host
> table
> 2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
> table
> 2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
> (AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
> 2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
> 2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.SecStorageSetupCommand: success
> 2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host
> table
> 2014-07-15 10:41:47,946 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed secondary storage
> nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
> 2014-07-15 10:41:47,968 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
> }
> 2014-07-15 10:41:47,976 DEBUG [agent.transport.Request]
> (AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:47,977 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:47,977 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed firewall rules into
> SSVM s-909-VM
> 2014-07-15 10:41:47,985 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
> }
> 2014-07-15 10:41:48,016 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"storage.ListVolumeCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","wait":0}}] }
> 2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
> (AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:48,033 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed firewall rules into
> s-889-VM
> 2014-07-15 10:41:48,059 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","
> 10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","
> 192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
> }
> 2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
> (AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"storage.ListVolumeAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
> 2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
> 2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.storage.ListVolumeCommand: success
> 2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Monitor DownloadListener says there is an
> error in the connect process for 42 due to null
> java.lang.NullPointerException
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
> at
> com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
> at
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Host 42 is disconnecting with event
> AgentDisconnected
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) The next status of agent 42is Alert, current
> status is Connecting
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Remove Agent : 42
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache]
> (AgentManager-Handler-2:null) Processing Disconnect.
> 2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.security.SecurityGroupListener
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.listener.StoragePoolMonitor
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.download.DownloadListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.upload.UploadListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.capacity.StorageCapacityListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.capacity.ComputeCapacityListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.consoleproxy.ConsoleProxyListener
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.SshKeysDistriMonitor
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl]
> (AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
> 2014-07-15 10:41:48,101 DEBUG [cloud.host.Status]
> (AgentManager-Handler-2:null) Transition:[Resource state = Enabled, Agent
> event = AgentDisconnected, Host id = 42, name = s-909-VM]
> 2014-07-15 10:41:48,163 DEBUG [cloud.host.Status]
> (AgentManager-Handler-2:null) Agent status update: [id = 42; name =
> s-909-VM; old status = Connecting; event = AgentDisconnected; new status =
> Alert; old update count = 209; new update count = 210]
> 2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-2:null) Notifying other nodes of to disconnect
> 2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Can not send command
> com.cloud.agent.api.ReadyCommand due to Host 42 is not up
> 2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Unable to create attache for agent: Seq
> 42-208: { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101,
> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}]
> }
> 2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"Answer":{"result":true,"details":"","wait":0}}] }
> 2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
> 2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
> 2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.SecStorageVMSetupCommand:
> 2014-07-15 10:41:48,208 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed http auth into
> s-889-VM
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
> 2014-07-15 10:41:48,227 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"storage.ListTemplateCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","wait":0}}] }
> 2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
> (AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"storage.ListTemplateAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
> 2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
> 2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.storage.ListTemplateCommand: success
> 2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found routing-3 already in the
> template host table
> 2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already
> in the template host table
> 2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync did not find
> centos56-x86_64-xen ready on server 2, will request download to
> start/resume shortly
> 2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready
> on server 2, will request download to start/resume shortly
> 2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host
> table
> 2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host
> table
> 2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
> 2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
> 2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
> table
> 2014-07-15 10:41:48,922 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"storage.ListVolumeCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","wait":0}}] }
> 2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
> (AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"storage.ListVolumeAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
> 2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
> 2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.storage.ListVolumeCommand: success
> 2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Monitor DownloadListener says there is an
> error in the connect process for 41 due to null
> java.lang.NullPointerException
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
> at
> com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
> at
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Host 41 is disconnecting with event
> AgentDisconnected
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) The next status of agent 41is Alert, current
> status is Connecting
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Remove Agent : 41
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache]
> (AgentManager-Handler-7:null) Processing Disconnect.
> 2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.security.SecurityGroupListener
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.listener.StoragePoolMonitor
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.download.DownloadListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.upload.UploadListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.capacity.StorageCapacityListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.capacity.ComputeCapacityListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.consoleproxy.ConsoleProxyListener
> 2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.SshKeysDistriMonitor
> 2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
> 2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl]
> (AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
> 2014-07-15 10:41:48,964 DEBUG [cloud.host.Status]
> (AgentManager-Handler-7:null) Transition:[Resource state = Enabled, Agent
> event = AgentDisconnected, Host id = 41, name = s-889-VM]
> 2014-07-15 10:41:49,006 DEBUG [cloud.host.Status]
> (AgentManager-Handler-7:null) Agent status update: [id = 41; name =
> s-889-VM; old status = Connecting; event = AgentDisconnected; new status =
> Alert; old update count = 391; new update count = 392]
> 2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) Notifying other nodes of to disconnect
> 2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Can not send command
> com.cloud.agent.api.ReadyCommand due to Host 41 is not up
> 2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Unable to create attache for agent: Seq
> 41-390: { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101,
> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}]
> }
> 2014-07-15 10:41:50,555 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1
> is ready to launch secondary storage VM
> 2014-07-15 10:41:50,609 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 2
> is ready to launch secondary storage VM
> 2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> 2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 2 is ready to launch console proxy
> 2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) StorageCollector is running...
> 2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.18.7/export/d2secondary
> 2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.18.7/export/d3secondary
> 2014-07-15 10:41:51,928 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers.
> 2014-07-15 10:41:51,991 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 7-513215195: Received: { Ans: , MgmtId:
> 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,059 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 9-1410599594: Received: { Ans: , MgmtId:
> 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,128 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 14-1320747056: Received: { Ans: , MgmtId:
> 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,150 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 16-940116823: Received: { Ans: , MgmtId:
> 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,231 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 14-1320747057: Received: { Ans: , MgmtId:
> 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,312 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 7-513215196: Received: { Ans: , MgmtId:
> 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-6:null) Ping from 23
> 2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-13:null) Ping from 8
> 2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-5:null) Ping from 40
>
> Disclaimer:
> The contents of this e-mail, and its attachments, if any, are confidential
> and may be protected
> by law against any unauthorized use. If you have received this e-mail by
> mistake or have
> reason to believe that you are not the intended recipient, please notify
> the sender by reply
> e-mail as soon as possible and delete it from your computer system
> immediately thereafter.
> If you are not the intended recipient, you must not copy this e-mail or
> attachment or disclose
> the contents to any other person. While we have made every effort to keep
> our network virus free,
> we take no responsibility for any computer virus which might be
> transferred by way of this e-mail.
>
Re: 回复：RE: 注册模板后出现状态：Storage agent or storage VM disconnected
Posted by Wei ZHOU <us...@gmail.com>.
你看一下ssvm里面的log吧
/var/log/cloud/目录
2014-07-15 10:34 GMT+02:00 赵涛 <zh...@qq.com>:
> 在vm_instance表里 s-909-VM/ s-889-VM是Running ，但在
> host表里显示状态是Alerting，销毁重建后还是一样
> ------------------ 原始邮件 ------------------
> 发件人: "YUAN, GUI -HBIS";<yu...@hbis.com.cn>;
> 发送时间: 2014年7月15日(星期二) 下午2:58
> 收件人: "users-cn"<us...@cloudstack.apache.org>;
>
> 主题: RE: 注册模板后出现状态：Storage agent or storage VM disconnected
>
>
>
> 1.SSVM没有启动，可能是对应的AgentHost连不上。确认有可用的HOST
> 如：s-909-VM/ s-889-VM
>
> 2.如果SSVM正常，模板添加时，提示SSVM- response time out
> 则需要确认NFS服务器的NFS服务状态及防火墙设置
>
>
>
> -----Original Message-----
> From: 赵涛 [mailto:zhaotao_cn@qq.com]
> Sent: Tuesday, July 15, 2014 12:46 PM
> To: users-cn
> Subject: 注册模板后出现状态：Storage agent or storage VM disconnected
>
> 注册模板后出现状态：Storage agent or storage VM disconnected,删除SSVM后问题依旧，请大神指导
>
> 删除SSVM虚拟机后重建的SSVM虚拟机上报日志：
>
>
> 2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> SecurityGroupListener
> 2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener]
> (AgentManager-Handler-7:null) Received a host startup notification
> 2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> StoragePoolMonitor
> 2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> SecondaryStorageListener
> 2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener]
> (AgentManager-Handler-7:null) Received a host startup notification
> com.cloud.agent.api.StartupSecondaryStorageCommand
> 2014-07-15 10:41:46,714 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}]
> }
> 2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-14:null) Ping from 41
> 2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
> (AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, {
> SecStorageSetupAnswer } }
> 2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
> 2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.SecStorageSetupCommand: success
> 2014-07-15 10:41:46,809 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed secondary storage
> nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
> 2014-07-15 10:41:46,831 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
> }
> 2014-07-15 10:41:46,880 DEBUG [agent.transport.Request]
> (AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:46,881 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:46,881 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed firewall rules into
> SSVM s-889-VM
> 2014-07-15 10:41:46,891 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
> }
> 2014-07-15 10:41:46,901 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:46,902 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:46,902 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed firewall rules into
> s-909-VM
> 2014-07-15 10:41:46,929 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","
> 10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","
> 192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
> }
> 2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
> (AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"Answer":{"result":true,"details":"","wait":0}}] }
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
> 2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.SecStorageVMSetupCommand:
> 2014-07-15 10:41:47,083 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-2:null) Successfully programmed http auth into
> s-909-VM
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener:
> NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener:
> AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
> 2014-07-15 10:41:47,101 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"storage.ListTemplateCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","wait":0}}] }
> 2014-07-15 10:41:47,629 DEBUG [agent.transport.Request]
> (AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"storage.ListTemplateAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
> 2014-07-15 10:41:47,630 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer
> } }
> 2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.storage.ListTemplateCommand: success
> 2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found routing-3 already in the
> template host table
> 2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already
> in the template host table
> 2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync did not find
> centos56-x86_64-xen ready on server 11, will request download to
> start/resume shortly
> 2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready
> on server 11, will request download to start/resume shortly
> 2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host
> table
> 2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host
> table
> 2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
> table
> 2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
> (AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
> 2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
> 2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.SecStorageSetupCommand: success
> 2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-2:null) Template Sync found
> 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host
> table
> 2014-07-15 10:41:47,946 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed secondary storage
> nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
> 2014-07-15 10:41:47,968 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
> }
> 2014-07-15 10:41:47,976 DEBUG [agent.transport.Request]
> (AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:47,977 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:47,977 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed firewall rules into
> SSVM s-909-VM
> 2014-07-15 10:41:47,985 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100011,
> [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
> }
> 2014-07-15 10:41:48,016 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId:
> 73548179740843, via: 42, Ver: v1, Flags: 100111,
> [{"storage.ListVolumeCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","wait":0}}] }
> 2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
> (AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10,
> [{"Answer":{"result":true,"wait":0}}] }
> 2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
> 2014-07-15 10:41:48,033 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed firewall rules into
> s-889-VM
> 2014-07-15 10:41:48,059 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","
> 10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","
> 192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
> }
> 2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
> (AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110,
> [{"storage.ListVolumeAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
> 2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
> (AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: ,
> MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
> 2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Details from executing class
> com.cloud.agent.api.storage.ListVolumeCommand: success
> 2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Monitor DownloadListener says there is an
> error in the connect process for 42 due to null
> java.lang.NullPointerException
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
> at
> com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
> at
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Host 42 is disconnecting with event
> AgentDisconnected
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) The next status of agent 42is Alert, current
> status is Connecting
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Remove Agent : 42
> 2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache]
> (AgentManager-Handler-2:null) Processing Disconnect.
> 2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.security.SecurityGroupListener
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.listener.StoragePoolMonitor
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.download.DownloadListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.upload.UploadListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.capacity.StorageCapacityListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.capacity.ComputeCapacityListener
> 2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.consoleproxy.ConsoleProxyListener
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.SshKeysDistriMonitor
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
> 2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Sending Disconnect to listener:
> com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl]
> (AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
> 2014-07-15 10:41:48,101 DEBUG [cloud.host.Status]
> (AgentManager-Handler-2:null) Transition:[Resource state = Enabled, Agent
> event = AgentDisconnected, Host id = 42, name = s-909-VM]
> 2014-07-15 10:41:48,163 DEBUG [cloud.host.Status]
> (AgentManager-Handler-2:null) Agent status update: [id = 42; name =
> s-909-VM; old status = Connecting; event = AgentDisconnected; new status =
> Alert; old update count = 209; new update count = 210]
> 2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-2:null) Notifying other nodes of to disconnect
> 2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Can not send command
> com.cloud.agent.api.ReadyCommand due to Host 42 is not up
> 2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-2:null) Unable to create attache for agent: Seq
> 42-208: { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101,
> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}]
> }
> 2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"Answer":{"result":true,"details":"","wait":0}}] }
> 2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
> 2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
> 2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.SecStorageVMSetupCommand:
> 2014-07-15 10:41:48,208 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl]
> (AgentManager-Handler-7:null) Successfully programmed http auth into
> s-889-VM
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener:
> AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
> 2014-07-15 10:41:48,227 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"storage.ListTemplateCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","wait":0}}] }
> 2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
> (AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"storage.ListTemplateAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
> 2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
> 2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.storage.ListTemplateCommand: success
> 2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found routing-3 already in the
> template host table
> 2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already
> in the template host table
> 2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync did not find
> centos56-x86_64-xen ready on server 2, will request download to
> start/resume shortly
> 2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready
> on server 2, will request download to start/resume shortly
> 2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host
> table
> 2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host
> table
> 2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
> 2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
> 2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl]
> (AgentManager-Handler-7:null) Template Sync found
> 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host
> table
> 2014-07-15 10:41:48,922 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 100111,
> [{"storage.ListVolumeCommand":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","wait":0}}] }
> 2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
> (AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: ,
> MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110,
> [{"storage.ListVolumeAnswer":{"secUrl":"nfs://
> 10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
> }
> 2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
> 2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId:
> 73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
> 2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Details from executing class
> com.cloud.agent.api.storage.ListVolumeCommand: success
> 2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Monitor DownloadListener says there is an
> error in the connect process for 41 due to null
> java.lang.NullPointerException
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
> at
> com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
> at
> com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
> at
> com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Host 41 is disconnecting with event
> AgentDisconnected
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) The next status of agent 41is Alert, current
> status is Connecting
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Remove Agent : 41
> 2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache]
> (AgentManager-Handler-7:null) Processing Disconnect.
> 2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.security.SecurityGroupListener
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.listener.StoragePoolMonitor
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
> 2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.download.DownloadListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.upload.UploadListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.capacity.StorageCapacityListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.capacity.ComputeCapacityListener
> 2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.consoleproxy.ConsoleProxyListener
> 2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.SshKeysDistriMonitor
> 2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
> 2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Sending Disconnect to listener:
> com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl]
> (AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
> 2014-07-15 10:41:48,964 DEBUG [cloud.host.Status]
> (AgentManager-Handler-7:null) Transition:[Resource state = Enabled, Agent
> event = AgentDisconnected, Host id = 41, name = s-889-VM]
> 2014-07-15 10:41:49,006 DEBUG [cloud.host.Status]
> (AgentManager-Handler-7:null) Agent status update: [id = 41; name =
> s-889-VM; old status = Connecting; event = AgentDisconnected; new status =
> Alert; old update count = 391; new update count = 392]
> 2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) Notifying other nodes of to disconnect
> 2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Can not send command
> com.cloud.agent.api.ReadyCommand due to Host 41 is not up
> 2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-7:null) Unable to create attache for agent: Seq
> 41-390: { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101,
> [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}]
> }
> 2014-07-15 10:41:50,555 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1
> is ready to launch secondary storage VM
> 2014-07-15 10:41:50,609 DEBUG
> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 2
> is ready to launch secondary storage VM
> 2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> 2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 2 is ready to launch console proxy
> 2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) StorageCollector is running...
> 2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.18.7/export/d2secondary
> 2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) There is no secondary storage VM for secondary
> storage host nfs://10.11.18.7/export/d3secondary
> 2014-07-15 10:41:51,928 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers.
> 2014-07-15 10:41:51,991 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 7-513215195: Received: { Ans: , MgmtId:
> 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,059 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 9-1410599594: Received: { Ans: , MgmtId:
> 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,128 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 14-1320747056: Received: { Ans: , MgmtId:
> 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,150 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 16-940116823: Received: { Ans: , MgmtId:
> 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,231 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 14-1320747057: Received: { Ans: , MgmtId:
> 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:52,312 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 7-513215196: Received: { Ans: , MgmtId:
> 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-6:null) Ping from 23
> 2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-13:null) Ping from 8
> 2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-5:null) Ping from 40
>
> Disclaimer:
> The contents of this e-mail, and its attachments, if any, are confidential
> and may be protected
> by law against any unauthorized use. If you have received this e-mail by
> mistake or have
> reason to believe that you are not the intended recipient, please notify
> the sender by reply
> e-mail as soon as possible and delete it from your computer system
> immediately thereafter.
> If you are not the intended recipient, you must not copy this e-mail or
> attachment or disclose
> the contents to any other person. While we have made every effort to keep
> our network virus free,
> we take no responsibility for any computer virus which might be
> transferred by way of this e-mail.
>
回复：RE: 注册模板后出现状态：Storage agent or storage VM disconnected
Posted by 赵涛 <zh...@qq.com>.
在vm_instance表里 s-909-VM/ s-889-VM是Running ，但在
host表里显示状态是Alerting，销毁重建后还是一样
------------------ 原始邮件 ------------------
发件人: "YUAN, GUI -HBIS";<yu...@hbis.com.cn>;
发送时间: 2014年7月15日(星期二) 下午2:58
收件人: "users-cn"<us...@cloudstack.apache.org>;
主题: RE: 注册模板后出现状态：Storage agent or storage VM disconnected
1.SSVM没有启动，可能是对应的AgentHost连不上。确认有可用的HOST
如：s-909-VM/ s-889-VM
2.如果SSVM正常，模板添加时，提示SSVM- response time out
则需要确认NFS服务器的NFS服务状态及防火墙设置
-----Original Message-----
From: 赵涛 [mailto:zhaotao_cn@qq.com]
Sent: Tuesday, July 15, 2014 12:46 PM
To: users-cn
Subject: 注册模板后出现状态：Storage agent or storage VM disconnected
注册模板后出现状态：Storage agent or storage VM disconnected,删除SSVM后问题依旧，请大神指导
删除SSVM虚拟机后重建的SSVM虚拟机上报日志：
2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecurityGroupListener
2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener] (AgentManager-Handler-7:null) Received a host startup notification
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: StoragePoolMonitor
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecondaryStorageListener
2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener] (AgentManager-Handler-7:null) Received a host startup notification com.cloud.agent.api.StartupSecondaryStorageCommand
2014-07-15 10:41:46,714 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}] }
2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) Ping from 41
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:46,809 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
2014-07-15 10:41:46,831 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:46,880 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,881 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,881 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into SSVM s-889-VM
2014-07-15 10:41:46,891 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:46,901 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,902 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,902 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into s-909-VM
2014-07-15 10:41:46,929 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:47,083 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed http auth into s-909-VM
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:47,101 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:47,629 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
2014-07-15 10:41:47,630 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos56-x86_64-xen ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host table
2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host table
2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host table
2014-07-15 10:41:47,946 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
2014-07-15 10:41:47,968 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:47,976 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:47,977 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:47,977 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into SSVM s-909-VM
2014-07-15 10:41:47,985 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:48,016 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:48,033 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into s-889-VM
2014-07-15 10:41:48,059 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Monitor DownloadListener says there is an error in the connect process for 42 due to null java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Host 42 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) The next status of agent 42is Alert, current status is Connecting
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Remove Agent : 42
2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-2:null) Processing Disconnect.
2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
2014-07-15 10:41:48,101 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 42, name = s-909-VM]
2014-07-15 10:41:48,163 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Agent status update: [id = 42; name = s-909-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 209; new update count = 210]
2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Notifying other nodes of to disconnect
2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 42 is not up
2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Unable to create attache for agent: Seq 42-208: { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:48,208 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed http auth into s-889-VM
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:48,227 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos56-x86_64-xen ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host table
2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host table
2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:48,922 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Monitor DownloadListener says there is an error in the connect process for 41 due to null java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Host 41 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) The next status of agent 41is Alert, current status is Connecting
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Remove Agent : 41
2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-7:null) Processing Disconnect.
2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
2014-07-15 10:41:48,964 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 41, name = s-889-VM]
2014-07-15 10:41:49,006 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Agent status update: [id = 41; name = s-889-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 391; new update count = 392]
2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Notifying other nodes of to disconnect
2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 41 is not up
2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Unable to create attache for agent: Seq 41-390: { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:50,555 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-07-15 10:41:50,609 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 2 is ready to launch secondary storage VM
2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 2 is ready to launch console proxy
2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d2secondary
2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d3secondary
2014-07-15 10:41:51,928 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers.
2014-07-15 10:41:51,991 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215195: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,059 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 9-1410599594: Received: { Ans: , MgmtId: 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,128 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747056: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,150 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 16-940116823: Received: { Ans: , MgmtId: 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,231 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747057: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,312 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215196: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 23
2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 8
2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 40
Disclaimer:
The contents of this e-mail, and its attachments, if any, are confidential and may be protected
by law against any unauthorized use. If you have received this e-mail by mistake or have
reason to believe that you are not the intended recipient, please notify the sender by reply
e-mail as soon as possible and delete it from your computer system immediately thereafter.
If you are not the intended recipient, you must not copy this e-mail or attachment or disclose
the contents to any other person. While we have made every effort to keep our network virus free,
we take no responsibility for any computer virus which might be transferred by way of this e-mail.
RE: 注册模板后出现状态：Storage agent or storage VM disconnected
Posted by "YUAN, GUI -HBIS" <yu...@hbis.com.cn>.
1.SSVM没有启动，可能是对应的AgentHost连不上。确认有可用的HOST
如：s-909-VM/ s-889-VM
2.如果SSVM正常，模板添加时，提示SSVM- response time out
则需要确认NFS服务器的NFS服务状态及防火墙设置
-----Original Message-----
From: 赵涛 [mailto:zhaotao_cn@qq.com]
Sent: Tuesday, July 15, 2014 12:46 PM
To: users-cn
Subject: 注册模板后出现状态：Storage agent or storage VM disconnected
注册模板后出现状态：Storage agent or storage VM disconnected,删除SSVM后问题依旧，请大神指导
删除SSVM虚拟机后重建的SSVM虚拟机上报日志：
2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecurityGroupListener
2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener] (AgentManager-Handler-7:null) Received a host startup notification
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: StoragePoolMonitor
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecondaryStorageListener
2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener] (AgentManager-Handler-7:null) Received a host startup notification com.cloud.agent.api.StartupSecondaryStorageCommand
2014-07-15 10:41:46,714 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}] }
2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) Ping from 41
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:46,809 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
2014-07-15 10:41:46,831 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:46,880 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,881 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,881 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into SSVM s-889-VM
2014-07-15 10:41:46,891 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:46,901 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,902 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,902 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into s-909-VM
2014-07-15 10:41:46,929 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:47,083 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed http auth into s-909-VM
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:47,101 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:47,629 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
2014-07-15 10:41:47,630 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos56-x86_64-xen ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host table
2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host table
2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host table
2014-07-15 10:41:47,946 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
2014-07-15 10:41:47,968 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:47,976 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:47,977 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:47,977 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into SSVM s-909-VM
2014-07-15 10:41:47,985 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:48,016 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:48,033 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into s-889-VM
2014-07-15 10:41:48,059 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Monitor DownloadListener says there is an error in the connect process for 42 due to null java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Host 42 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) The next status of agent 42is Alert, current status is Connecting
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Remove Agent : 42
2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-2:null) Processing Disconnect.
2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
2014-07-15 10:41:48,101 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 42, name = s-909-VM]
2014-07-15 10:41:48,163 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Agent status update: [id = 42; name = s-909-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 209; new update count = 210]
2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Notifying other nodes of to disconnect
2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 42 is not up
2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Unable to create attache for agent: Seq 42-208: { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:48,208 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed http auth into s-889-VM
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:48,227 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos56-x86_64-xen ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host table
2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host table
2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:48,922 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Monitor DownloadListener says there is an error in the connect process for 41 due to null java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Host 41 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) The next status of agent 41is Alert, current status is Connecting
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Remove Agent : 41
2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-7:null) Processing Disconnect.
2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
2014-07-15 10:41:48,964 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 41, name = s-889-VM]
2014-07-15 10:41:49,006 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Agent status update: [id = 41; name = s-889-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 391; new update count = 392]
2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Notifying other nodes of to disconnect
2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 41 is not up
2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Unable to create attache for agent: Seq 41-390: { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:50,555 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-07-15 10:41:50,609 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 2 is ready to launch secondary storage VM
2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 2 is ready to launch console proxy
2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d2secondary
2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d3secondary
2014-07-15 10:41:51,928 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers.
2014-07-15 10:41:51,991 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215195: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,059 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 9-1410599594: Received: { Ans: , MgmtId: 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,128 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747056: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,150 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 16-940116823: Received: { Ans: , MgmtId: 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,231 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747057: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,312 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215196: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 23
2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 8
2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 40
Disclaimer:
The contents of this e-mail, and its attachments, if any, are confidential and may be protected
by law against any unauthorized use. If you have received this e-mail by mistake or have
reason to believe that you are not the intended recipient, please notify the sender by reply
e-mail as soon as possible and delete it from your computer system immediately thereafter.
If you are not the intended recipient, you must not copy this e-mail or attachment or disclose
the contents to any other person. While we have made every effort to keep our network virus free,
we take no responsibility for any computer virus which might be transferred by way of this e-mail.
注册模板后出现状态：Storage agent or storage VM disconnected
Posted by 赵涛 <zh...@qq.com>.
注册模板后出现状态：Storage agent or storage VM disconnected,删除SSVM后问题依旧，请大神指导
删除SSVM虚拟机后重建的SSVM虚拟机上报日志：
2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecurityGroupListener
2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener] (AgentManager-Handler-7:null) Received a host startup notification
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: StoragePoolMonitor
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: SecondaryStorageListener
2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener] (AgentManager-Handler-7:null) Received a host startup notification com.cloud.agent.api.StartupSecondaryStorageCommand
2014-07-15 10:41:46,714 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}] }
2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) Ping from 41
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:46,809 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
2014-07-15 10:41:46,831 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:46,880 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,881 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,881 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into SSVM s-889-VM
2014-07-15 10:41:46,891 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:46,901 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,902 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,902 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed firewall rules into s-909-VM
2014-07-15 10:41:46,929 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:47,083 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-2:null) Successfully programmed http auth into s-909-VM
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:47,101 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:47,629 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
2014-07-15 10:41:47,630 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos56-x86_64-xen ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host table
2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host table
2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-2:null) Template Sync found 275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host table
2014-07-15 10:41:47,946 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed secondary storage nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
2014-07-15 10:41:47,968 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}] }
2014-07-15 10:41:47,976 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:47,977 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:47,977 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into SSVM s-909-VM
2014-07-15 10:41:47,985 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100011, [{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}] }
2014-07-15 10:41:48,016 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:48,033 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed firewall rules into s-889-VM
2014-07-15 10:41:48,059 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: , MgmtId: 73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Monitor DownloadListener says there is an error in the connect process for 42 due to null
java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Host 42 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) The next status of agent 42is Alert, current status is Connecting
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Remove Agent : 42
2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-2:null) Processing Disconnect.
2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
2014-07-15 10:41:48,101 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 42, name = s-909-VM]
2014-07-15 10:41:48,163 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null) Agent status update: [id = 42; name = s-909-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 209; new update count = 210]
2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Notifying other nodes of to disconnect
2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 42 is not up
2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Unable to create attache for agent: Seq 42-208: { Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:48,208 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (AgentManager-Handler-7:null) Successfully programmed http auth into s-889-VM
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:48,227 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found routing-3 already in the template host table
2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already in the template host table
2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos56-x86_64-xen ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host table
2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host table
2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl] (AgentManager-Handler-7:null) Template Sync found 221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:48,922 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 100111, [{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, [{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}] }
2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId: 73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Details from executing class com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Monitor DownloadListener says there is an error in the connect process for 41 due to null
java.lang.NullPointerException
at com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Host 41 is disconnecting with event AgentDisconnected
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) The next status of agent 41is Alert, current status is Connecting
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Remove Agent : 41
2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache] (AgentManager-Handler-7:null) Processing Disconnect.
2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
2014-07-15 10:41:48,964 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 41, name = s-889-VM]
2014-07-15 10:41:49,006 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null) Agent status update: [id = 41; name = s-889-VM; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 391; new update count = 392]
2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Notifying other nodes of to disconnect
2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Can not send command com.cloud.agent.api.ReadyCommand due to Host 41 is not up
2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl] (AgentManager-Handler-7:null) Unable to create attache for agent: Seq 41-390: { Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101, [{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}] }
2014-07-15 10:41:50,555 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-07-15 10:41:50,609 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 2 is ready to launch secondary storage VM
2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 2 is ready to launch console proxy
2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d2secondary
2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://10.11.18.7/export/d3secondary
2014-07-15 10:41:51,928 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers.
2014-07-15 10:41:51,991 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215195: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,059 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 9-1410599594: Received: { Ans: , MgmtId: 73548179740843, via: 9, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,128 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747056: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,150 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 16-940116823: Received: { Ans: , MgmtId: 73548179740843, via: 16, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,231 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-1320747057: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,312 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-513215196: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 23
2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 8
2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 40