[HELP] Unable to create/register new templates

[Indra Pramana <in...@sg.or.id>]

Dear all,

I am running CloudStack version 4.2.0 with KVM hypervisor, Ceph RBD primary
storage and NFS secondary storage.

Since yesterday, I am not able to create/register a new template. Normally
I would be converting a root-disk volume into a template. In this case, the
process of creating the template will take forever and will never complete.

On management-server log, there is not much information other than a line
stating that the template is being copied:

====
2013-12-03 17:43:33,340 DEBUG [agent.transport.Request]
(Job-Executor-33:job-5446 = [ f6f014b3-7301-4be0-98e9-e6a8caba5e51 ]) Seq
37-2044592856: Sending  { Cmd , MgmtId: 161342671900, via: 37, Ver: v1,
Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"10c780a0-d574-4332-bcf4-9988e7ed963d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d433809b-01ea-3947-ba0f-48077244e4d6","id":214,"poolType":"RBD","host":"
xxxxxx.xx.com","path":"xxxxxx-xx-01","port":6789}},"name":"ROOT-281","size":10737418240,"path":"d9d5f42f-0c57-4b17-a093-f1172ba24b81","volumeId":306,"vmName":"i-2-281-VM","accountId":2,"format":"RAW","id":306,"hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/345","uuid":"73732183-69f7-47de-b2e8-556326ca989b","id":345,"format":"RAW","accountId":2,"hvm":true,"displayText":"CentOS
5.10 (64-bit)
20131203a","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://x.x.x.x/mnt/vol1/sec-storage2","_role":"Image"}},"name":"2c8002031-3971-33af-b59f-6f7b52011437","hypervisorType":"KVM"}},"executeInSequence":true,"wait":10800}}]
}
====

Logs from SSVM /var/log/cloud/cloud.out:

====
2013-12-03 11:18:58,137 WARN  [storage.template.DownloadManagerImpl]
(agentRequest-Handler-3:null) Post download installation was not completed
for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/343
2013-12-03 11:18:58,159 WARN  [storage.template.DownloadManagerImpl]
(agentRequest-Handler-3:null) Post download installation was not completed
for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/276
2013-12-03 11:18:58,186 WARN  [storage.template.DownloadManagerImpl]
(agentRequest-Handler-3:null) Post download installation was not completed
for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/275
2013-12-03 11:18:58,220 WARN  [storage.template.DownloadManagerImpl]
(agentRequest-Handler-3:null) Post download installation was not completed
for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/277
2013-12-03 11:18:58,234 WARN  [storage.template.DownloadManagerImpl]
(agentRequest-Handler-3:null) Post download installation was not completed
for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/225
====

After a while, the process will block other processes, i.e. I will not be
able to create a new VM or start/stop any VMs, it seems that it's waiting
for the template creation process to complete (which never completes). So I
would need to restart the cloudstack-management service on the management
server for the day-to-day operation to continue.

Status of the created template would be:

Ready: No
Status: (empty)
Size: (empty)

I have done all the SSVM health checks as mentioned on this URL:

https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting

And I can confirm that the SSVM is running fine, health checks passed. I
have tried to restart the SSVM service (service cloud restart) and even
stop and start the SSVM but the problem still persists and the templates
will never get created.

Any advice on how I can further troubleshoot this problem?

Looking forward to your reply, thank you.

Cheers.