You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/07/22 21:35:39 UTC
[GitHub] [cloudstack] tampler opened a new issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
tampler opened a new issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Secondary Storage, NFS, SystemVM
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.15.1
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Basic Zone
KVM hypervisor
Single host (management + agent)
NFS export
```
/export *(rw,async,no_root_squash,no_subtree_check)
/mnt/secondary *(rw,async,no_root_squash,no_subtree_check)
```
I also validated NFS functionality by reading/writing to the shared NFS folders from a different host
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
Ubuntu 20.04.2
KVM Hypervisor
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Related to #4022
After setting up a Basic Zone with a single host (management + agent), I see host resources, but no Secondary Storage available for CS.
I'm using a NFS backed Primary and Secondary Storage and set it up according to [this](https://rohityadav.cloud/blog/cloudstack-kvm/) manual.
I installed the SystemVM template with the following command:
```
wget http://packages.shapeblue.com/systemvmtemplate/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /export/secondary -f systemvmtemplate-4.15.1-kvm.qcow2.bz2 -h kvm \
-o localhost -r cloud -d cloud
```
Here's the log exert from management-server.log
```
2021-07-23 00:01:38,823 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-b1d45b1b) (logid:43c37cbc) Zone 1 is ready to launch secondary storage VM
2021-07-23 00:01:38,833 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-b1d45b1b) (logid:43c37cbc) No running secondary storage vms found in datacenter id=1, starting one
2021-07-23 00:01:38,848 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-b1d45b1b) (logid:43c37cbc) Found a stopped secondary storage vm, starting it. Vm id : 149
```
Full log here
[out.log](https://github.com/apache/cloudstack/files/6865325/out.log)
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
- NFS SecondaryStorage is available
- Can launch Secondary Storage VM and other guest VMs
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
1. Secondary Storage is unavailable ( 0.00 GB / 0.00 GB )
2. SystemVMs won't start with the exception: `2021-07-23 00:05:20,983 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-794f66d0) (logid:37fa38c9) Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-155-VM]Scope=interface com.cloud.dc.DataCenter; id=1`
3. Can't launch any guest VM
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tampler closed issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
Posted by GitBox <gi...@apache.org>.
tampler closed issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tampler commented on issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
Posted by GitBox <gi...@apache.org>.
tampler commented on issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238#issuecomment-885607411
@slavkap
Oooh, thanks a lot for your advice! I indeed somehow missed setting up the AppArmor during installation.
Now SecondaryStorageVM have launched and I see the secondary storage available and is being used in the Dashboard.
Thanks everyone for your help. Closing this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tampler commented on issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
Posted by GitBox <gi...@apache.org>.
tampler commented on issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238#issuecomment-885458030
Hello @DaanHoogland
1. Nah, SeLinux wasn't installed or enabled. I'm using a fresh Ubuntu 20.04.2 with default settings
2. ufw is inactive
Here's my net config:
```
root@cloud:~# ip r
default via 185.185.58.1 dev cloudbr0 proto static
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
185.185.58.0/26 dev cloudbr0 proto kernel scope link src 185.185.58.54
```
Here's my `agent.properties`:
```
#Fri Jul 23 00:28:34 MSK 2021
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
router.aggregation.command.each.timeout=600
guest.network.device=cloudbr0
hypervisor.type=kvm
public.network.device=cloudbr0
port=8250
zone=1
local.storage.uuid=acfa6fdc-259f-4608-b1c1-d99d11288164
host=185.185.58.54@static
guid=76fcdbe8-1954-4888-a267-0d3c1f6c3043
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
```
3. Yes, getting an exception in `libvirt-apparmor`:
```
2021-07-23 00:00:05,569 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Caught libvirt exception while fetching host information
org.libvirt.LibvirtException: invalid connection pointer in virConnectGetCapabilities
```
The full log is below
[ag.log](https://github.com/apache/cloudstack/files/6867218/ag.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238#issuecomment-885450165
@tampler , the first few lines of your log indicate a problem on your host:
```
2021-07-23 00:29:14,275 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-02a49881 job-618/job-619 ctx-095c008f) (logid:a4fec8a9) Seq 1-2628976282477527051: Received: { Ans: , MgmtId: 165101265622802, via: 1(cloud), Ver: v1, Flags: 10, { StartAnswer, Answer } }
2021-07-23 00:29:14,561 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-02a49881 job-618/job-619 ctx-095c008f) (logid:a4fec8a9) Unable to start VM on Host[-1-Routing] due to internal error: cannot load AppArmor profile 'libvirt-cafb7dd0-20cd-4935-85c6-20222928be8e'
2021-07-23 00:29:14,661 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-02a49881 job-618/job-619 ctx-095c008f) (logid:a4fec8a9) Cleaning up resources for the vm VM[SecondaryStorageVm|s-199-VM] in Starting state
```
Selinux involved? Can you look at your host what the agent log says?
the SSVM can't be started on the host due to profile AppArmor not being loaded...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] slavkap commented on issue #5238: NFS backed SecondaryStorage is unavailable after Basic Zone setup
Posted by GitBox <gi...@apache.org>.
slavkap commented on issue #5238:
URL: https://github.com/apache/cloudstack/issues/5238#issuecomment-885468914
@tampler, the problem now is not with the secondary storage VMs, I guess your host state is not `Up`. According to CS documentation, the Apparmor has to be disabled [Security Policies](https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#configure-the-security-policies). Can you try with this setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org