You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Yordan Kostov <Yo...@NSOGROUP.COM> on 2021/05/05 10:14:35 UTC
cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan
Re: cannot create template?
Posted by Harikrishna Patnala <Ha...@shapeblue.com>.
Hi Jordan,
Glad that it helped to find the problem.
To know exactly what is happening when a command runs via vmops, from cloud.log you can see that command is initiated something like "DEBUG [root] #### CLOUD enter delete_secondary_storage_folder ####" and you can look for the command definition in vmops or vmopsSnapshots scripts under /etc/xapi.d/plugins.
Regards,
Harikrishna
________________________________
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Friday, May 7, 2021 7:38 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: cannot create template?
NFS is now able to mount for template provisioning.
Updated the XCPNG hosts and reboot and voila it is working!
So little thing to cause so much trouble.
Thank you for the tips Harikrishna!
By the way how do you find what command Cloudstack is using at such occasions?
It is a nice approach for troubleshooting purposes.
Regards,
Jordan
-----Original Message-----
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Friday, May 7, 2021 11:10 AM
To: users@cloudstack.apache.org
Subject: RE: cannot create template?
[X] This message came from outside your organization
Thank you for the tip!
This command actually those not work - "mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
Verbose output reports:
mount.nfs: timeout set for Fri May 7 09:58:35 2021
mount.nfs: trying text-based options 'soft,tcp,timeo=10,retrans=1,vers=4.1,addr=A.A.A.A,clientaddr=B.B.B.B'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed
My previous tests (and cloudstack mounts) that were working were all on NFSv3. This is very weird that certain Cloudstack mounts are made through NFSv3 and others through v4.
I do tried to mount nfs v4 from XenCenter and it failed the same way.
- there is nothing in the XCPNG logs - SMlog or xensource log
- there is no traffic coming to the NFS server nor any traffic is seen on the firewall.
I believe there is something wrong with the XCP and Nfs v4 mount request, as soon as execute the command with NFS v3 specified it works "mount -o soft,tcp,timeo=133,retrans=1, vers=3 <remote directory> <local directory>"
I will continue to investigate.
Regards,
Jordan
-----Original Message-----
From: Harikrishna Patnala <Ha...@shapeblue.com>
Sent: Thursday, May 6, 2021 5:43 AM
To: users@cloudstack.apache.org
Subject: Re: cannot create template?
[X] This message came from outside your organization
Hi Jordon,
You can try running the below mount command on XCP host which runs during this operation to see what is the actual mount problem, you can check it in verbose mode too.
"mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
The other error raises a concern of adding unsupported XCP 8.2 version is added properly in CloudStack. I see your previous email thread about XCP-ng 8.2 support.
"Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?"
Can you check around the above exception if you get any clue around why exactly it failed. You can try the same operations with some lower versions of XCP-ng and see if it fails there too.
Regards,
Harikrishna
________________________________
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Wednesday, May 5, 2021 6:17 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: cannot create template?
It seems the host tries to mount the NFS and move the template there.
Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed.
Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen.
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] create_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] #### CLOUD enter delete_secondary_storage_folder #### DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'} DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
DEBUG [root] delete_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
NFS firewall and XCPNG iptables are disabled.
Best regards,
Jordan
From: Yordan Kostov
Sent: Wednesday, May 5, 2021 1:15 PM
To: 'users@cloudstack.apache.org' <us...@cloudstack.apache.org>
Subject: cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan
<font size="2"><font color="#D8D8D8">11!</font>
RE: cannot create template?
Posted by Yordan Kostov <Yo...@NSOGROUP.COM>.
NFS is now able to mount for template provisioning.
Updated the XCPNG hosts and reboot and voila it is working!
So little thing to cause so much trouble.
Thank you for the tips Harikrishna!
By the way how do you find what command Cloudstack is using at such occasions?
It is a nice approach for troubleshooting purposes.
Regards,
Jordan
-----Original Message-----
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Friday, May 7, 2021 11:10 AM
To: users@cloudstack.apache.org
Subject: RE: cannot create template?
[X] This message came from outside your organization
Thank you for the tip!
This command actually those not work - "mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
Verbose output reports:
mount.nfs: timeout set for Fri May 7 09:58:35 2021
mount.nfs: trying text-based options 'soft,tcp,timeo=10,retrans=1,vers=4.1,addr=A.A.A.A,clientaddr=B.B.B.B'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed
My previous tests (and cloudstack mounts) that were working were all on NFSv3. This is very weird that certain Cloudstack mounts are made through NFSv3 and others through v4.
I do tried to mount nfs v4 from XenCenter and it failed the same way.
- there is nothing in the XCPNG logs - SMlog or xensource log
- there is no traffic coming to the NFS server nor any traffic is seen on the firewall.
I believe there is something wrong with the XCP and Nfs v4 mount request, as soon as execute the command with NFS v3 specified it works "mount -o soft,tcp,timeo=133,retrans=1, vers=3 <remote directory> <local directory>"
I will continue to investigate.
Regards,
Jordan
-----Original Message-----
From: Harikrishna Patnala <Ha...@shapeblue.com>
Sent: Thursday, May 6, 2021 5:43 AM
To: users@cloudstack.apache.org
Subject: Re: cannot create template?
[X] This message came from outside your organization
Hi Jordon,
You can try running the below mount command on XCP host which runs during this operation to see what is the actual mount problem, you can check it in verbose mode too.
"mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
The other error raises a concern of adding unsupported XCP 8.2 version is added properly in CloudStack. I see your previous email thread about XCP-ng 8.2 support.
"Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?"
Can you check around the above exception if you get any clue around why exactly it failed. You can try the same operations with some lower versions of XCP-ng and see if it fails there too.
Regards,
Harikrishna
________________________________
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Wednesday, May 5, 2021 6:17 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: cannot create template?
It seems the host tries to mount the NFS and move the template there.
Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed.
Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen.
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] create_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] #### CLOUD enter delete_secondary_storage_folder #### DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'} DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
DEBUG [root] delete_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
NFS firewall and XCPNG iptables are disabled.
Best regards,
Jordan
From: Yordan Kostov
Sent: Wednesday, May 5, 2021 1:15 PM
To: 'users@cloudstack.apache.org' <us...@cloudstack.apache.org>
Subject: cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan
<font size="2"><font color="#D8D8D8">11!</font>
RE: cannot create template?
Posted by Yordan Kostov <Yo...@NSOGROUP.COM>.
Thank you for the tip!
This command actually those not work - "mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
Verbose output reports:
mount.nfs: timeout set for Fri May 7 09:58:35 2021
mount.nfs: trying text-based options 'soft,tcp,timeo=10,retrans=1,vers=4.1,addr=A.A.A.A,clientaddr=B.B.B.B'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed
My previous tests (and cloudstack mounts) that were working were all on NFSv3. This is very weird that certain Cloudstack mounts are made through NFSv3 and others through v4.
I do tried to mount nfs v4 from XenCenter and it failed the same way.
- there is nothing in the XCPNG logs - SMlog or xensource log
- there is no traffic coming to the NFS server nor any traffic is seen on the firewall.
I believe there is something wrong with the XCP and Nfs v4 mount request, as soon as execute the command with NFS v3 specified it works
"mount -o soft,tcp,timeo=133,retrans=1, vers=3 <remote directory> <local directory>"
I will continue to investigate.
Regards,
Jordan
-----Original Message-----
From: Harikrishna Patnala <Ha...@shapeblue.com>
Sent: Thursday, May 6, 2021 5:43 AM
To: users@cloudstack.apache.org
Subject: Re: cannot create template?
[X] This message came from outside your organization
Hi Jordon,
You can try running the below mount command on XCP host which runs during this operation to see what is the actual mount problem, you can check it in verbose mode too.
"mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
The other error raises a concern of adding unsupported XCP 8.2 version is added properly in CloudStack. I see your previous email thread about XCP-ng 8.2 support.
"Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?"
Can you check around the above exception if you get any clue around why exactly it failed. You can try the same operations with some lower versions of XCP-ng and see if it fails there too.
Regards,
Harikrishna
________________________________
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Wednesday, May 5, 2021 6:17 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: cannot create template?
It seems the host tries to mount the NFS and move the template there.
Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed.
Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen.
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] create_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] #### CLOUD enter delete_secondary_storage_folder #### DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'} DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
DEBUG [root] delete_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
NFS firewall and XCPNG iptables are disabled.
Best regards,
Jordan
From: Yordan Kostov
Sent: Wednesday, May 5, 2021 1:15 PM
To: 'users@cloudstack.apache.org' <us...@cloudstack.apache.org>
Subject: cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan
Re: cannot create template?
Posted by Harikrishna Patnala <Ha...@shapeblue.com>.
Hi Jordon,
You can try running the below mount command on XCP host which runs during this operation to see what is the actual mount problem, you can check it in verbose mode too.
"mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>"
The other error raises a concern of adding unsupported XCP 8.2 version is added properly in CloudStack. I see your previous email thread about XCP-ng 8.2 support.
"Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?"
Can you check around the above exception if you get any clue around why exactly it failed. You can try the same operations with some lower versions of XCP-ng and see if it fails there too.
Regards,
Harikrishna
________________________________
From: Yordan Kostov <Yo...@NSOGROUP.COM>
Sent: Wednesday, May 5, 2021 6:17 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: cannot create template?
It seems the host tries to mount the NFS and move the template there.
Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed.
Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen.
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] create_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] #### CLOUD enter delete_secondary_storage_folder ####
DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'}
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
DEBUG [root] delete_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
NFS firewall and XCPNG iptables are disabled.
Best regards,
Jordan
From: Yordan Kostov
Sent: Wednesday, May 5, 2021 1:15 PM
To: 'users@cloudstack.apache.org' <us...@cloudstack.apache.org>
Subject: cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan
RE: cannot create template?
Posted by Yordan Kostov <Yo...@NSOGROUP.COM>.
It seems the host tries to mount the NFS and move the template there.
Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed.
Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen.
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] create_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003
DEBUG [root] #### CLOUD enter delete_secondary_storage_folder ####
DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'}
DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
DEBUG [root] delete_secondary_storage_folder failed.
DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014
NFS firewall and XCPNG iptables are disabled.
Best regards,
Jordan
From: Yordan Kostov
Sent: Wednesday, May 5, 2021 1:15 PM
To: 'users@cloudstack.apache.org' <us...@cloudstack.apache.org>
Subject: cannot create template?
Hey everyone,
Bumped into the following error today while trying to create a template from ROOT volume.
Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage
The process failed with the following error message in the management server logs:
callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin.
GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?
I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs.
Best Regards,
Jordan