NFS mount, snapshot and template issue - was Re: [HELP] Unable to create/register new templates

[Indra Pramana <in...@sg.or.id>]

Dear all,

Anyone can help on this? :)

Still related to this problem, I noted that our KVM hypervisors are having
issues mounting to the NFS secondary storage. Symptoms include "df" command
will hung (but not df -l), and we cannot change directory (cd) to the
mounted secondary storage.

I can confirm that the secondary storage is fine, I can mount from SSVM
just fine (but not the hosts). Mounting using o proto=tcp,port=2049 works
fine, but not using normal mount. Can I configure libvirt to use TCP
protocol when connecting to netfs?

Backup snapshot is also failing, with status "Created on Primary". This
means the snapshot is created on our primary storage (we are using Ceph RBD
primary storage), but failed to be copied over to the NFS secondary storage
due to the same NFS mount issue. agent.log shows that the snapshot is being
copied but when I checked the secondary storage, it was created with 0
bytes file size.

Excerpts from agent.log on the KVM hypervisor:

====
2013-12-06 05:11:39,904 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Didn't find an existing storage pool
c10404d3-070e-3579-980e-cb0d40effb7b by UUID, checking for pools with
duplicate paths
2013-12-06 05:11:39,907 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
22b0d26c-d8fa-490e-9382-9b4685e2b08e against pool we want to create
2013-12-06 05:11:39,912 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
d433809b-01ea-3947-ba0f-48077244e4d6 against pool we want to create
2013-12-06 05:11:39,917 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to create storage pool
c10404d3-070e-3579-980e-cb0d40effb7b
2013-12-06 05:11:39,921 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='netfs'>
<name>c10404d3-070e-3579-980e-cb0d40effb7b</name>
<uuid>c10404d3-070e-3579-980e-cb0d40effb7b</uuid>
<source>
<host name='x.x.x.x'/>
<dir path='/mnt/vol1/sec-storage2'/>
</source>
<target>
<path>/mnt/c10404d3-070e-3579-980e-cb0d40effb7b</path>
</target>
</pool>

2013-12-06 05:11:49,305 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
ceph-mon.simplercloud.com:6789
2013-12-06 05:11:49,335 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Attempting to create
/mnt/c10404d3-070e-3579-980e-cb0d40effb7b/snapshots/364/2847 recursively
2013-12-06 05:11:49,349 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Backing up RBD snapshot to
/mnt/c10404d3-070e-3579-980e-cb0d40effb7b/snapshots/364/2847/a5173b30-2bff-4927-b56c-c0db0f02e4cd
...
2013-12-06 05:16:00,780 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Completed backing up RBD snapshot
a5173b30-2bff-4927-b56c-c0db0f02e4cd to
/mnt/c10404d3-070e-3579-980e-cb0d40effb7b/snapshots/364/2847/a5173b3
0-2bff-4927-b56c-c0db0f02e4cd. Bytes written: 10737418240
=====

Based on the logs above, it seems that the backing up of the RBD snapshot
is completed, but in fact it's not. The status of the snapshot is
"CreatedOnPrimary", and when I log in to the secondary server to check, the
snapshot file is created but with 0 (zero) file size.

-rw-r--r--  1 root  wheel  0 Dec  6 05:11
a5173b30-2bff-4927-b56c-c0db0f02e4cd

I suspect that the NFS mount issue is the root cause of the problem, which
in turns affect the inability of creating/registering templates/ISO and
backup snapshot. Any advice on this is greatly appreciated!

Looking forward to your reply, thank you.

Cheers.



On Tue, Dec 3, 2013 at 9:14 PM, Indra Pramana <in...@sg.or.id> wrote:

> Dear all,
>
> I am running CloudStack version 4.2.0 with KVM hypervisor, Ceph RBD
> primary storage and NFS secondary storage.
>
> Since yesterday, I am not able to create/register a new template. Normally
> I would be converting a root-disk volume into a template. In this case, the
> process of creating the template will take forever and will never complete.
>
> On management-server log, there is not much information other than a line
> stating that the template is being copied:
>
> ====
> 2013-12-03 17:43:33,340 DEBUG [agent.transport.Request]
> (Job-Executor-33:job-5446 = [ f6f014b3-7301-4be0-98e9-e6a8caba5e51 ]) Seq
> 37-2044592856: Sending  { Cmd , MgmtId: 161342671900, via: 37, Ver: v1,
> Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"10c780a0-d574-4332-bcf4-9988e7ed963d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d433809b-01ea-3947-ba0f-48077244e4d6","id":214,"poolType":"RBD","host":"
> xxxxxx.xx.com","path":"xxxxxx-xx-01","port":6789}},"name":"ROOT-281","size":10737418240,"path":"d9d5f42f-0c57-4b17-a093-f1172ba24b81","volumeId":306,"vmName":"i-2-281-VM","accountId":2,"format":"RAW","id":306,"hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/345","uuid":"73732183-69f7-47de-b2e8-556326ca989b","id":345,"format":"RAW","accountId":2,"hvm":true,"displayText":"CentOS
> 5.10 (64-bit)
> 20131203a","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://x.x.x.x/mnt/vol1/sec-storage2","_role":"Image"}},"name":"2c8002031-3971-33af-b59f-6f7b52011437","hypervisorType":"KVM"}},"executeInSequence":true,"wait":10800}}]
> }
> ====
>
> Logs from SSVM /var/log/cloud/cloud.out:
>
> ====
> 2013-12-03 11:18:58,137 WARN  [storage.template.DownloadManagerImpl]
> (agentRequest-Handler-3:null) Post download installation was not completed
> for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/343
> 2013-12-03 11:18:58,159 WARN  [storage.template.DownloadManagerImpl]
> (agentRequest-Handler-3:null) Post download installation was not completed
> for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/276
> 2013-12-03 11:18:58,186 WARN  [storage.template.DownloadManagerImpl]
> (agentRequest-Handler-3:null) Post download installation was not completed
> for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/275
> 2013-12-03 11:18:58,220 WARN  [storage.template.DownloadManagerImpl]
> (agentRequest-Handler-3:null) Post download installation was not completed
> for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/277
> 2013-12-03 11:18:58,234 WARN  [storage.template.DownloadManagerImpl]
> (agentRequest-Handler-3:null) Post download installation was not completed
> for /mnt/SecStorage/41cab17f-4dbc-37a5-b8d2-cdc153301319/template/tmpl/2/225
> ====
>
> After a while, the process will block other processes, i.e. I will not be
> able to create a new VM or start/stop any VMs, it seems that it's waiting
> for the template creation process to complete (which never completes). So I
> would need to restart the cloudstack-management service on the management
> server for the day-to-day operation to continue.
>
> Status of the created template would be:
>
> Ready: No
> Status: (empty)
> Size: (empty)
>
> I have done all the SSVM health checks as mentioned on this URL:
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting
>
> And I can confirm that the SSVM is running fine, health checks passed. I
> have tried to restart the SSVM service (service cloud restart) and even
> stop and start the SSVM but the problem still persists and the templates
> will never get created.
>
> Any advice on how I can further troubleshoot this problem?
>
> Looking forward to your reply, thank you.
>
> Cheers.
>