You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "li-liwen (via GitHub)" <gi...@apache.org> on 2023/12/24 16:16:42 UTC
[I] Unable to attach Ceph as primary storage [cloudstack]
li-liwen opened a new issue, #8405:
URL: https://github.com/apache/cloudstack/issues/8405
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
UI
Primary storage
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.18.1.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Advanced networking
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
I am using Ubuntu server 22.04 as KVM hypervisor (Libvirt 8.0.0, QEMU 6.2.0) with ceph (18.2.1) installed on all hypervisors for a hyper-converged setup. The ceph cluster was bootstrapped by Cephadm.
The ceph cluster contains three separated monitor nodes , and five KVM+Ceph nodes. In the five KVM+Ceph nodes, two of them also function as monitor nodes.
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Cannot add ceph cluster as primary storage through Web UI.
I am also running a Proxmox cluster and I am able to add the ceph cluster in Proxmox with the same user and keys. In Proxmox, it works as expected.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
1. Install Ubuntu server 22.04 LTS on all machines.
2. Install cloudstack 4.18.1.0 management server as official document.
3. Bootstrape a Ceph Reef cluster through Cephadm (I have tried to install Cephadm with apt/curl, same result).
4. Install KVM and prepare networking on all KVM hosts.
5. Create cloudstack pool and init rbd
~~~
ceph osd pool create cloudstack
rbd pool init cloudstack
~~~
6. Create cloudstack user
~~~
ceph auth get-or-create client.cloudstack mon 'profile rbd' osd 'profile rbd pool=cloudstack'
~~~
Additionally, I also tried propagate the ceph.conf file and keyrings to each hosts but not working (even with admin keys).
I have read the thread #6463 and tried the following ceph configuration commands, but still no progress:
~~~
ceph config set mon auth_expose_insecure_global_id_reclaim false
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
ceph config set mon auth_allow_insecure_global_id_reclaim false
~~~
The KVM hosts can use the ceph cluster directly if provided the admin keyring. I used the following command:
~~~
qemu-img create -f rbd rbd:cloudstack/new-libvirt-image 2G
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
The ceph cluster is added to cloudstack successfully.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
This is the cloudstack troubleshooting log right after I click the OK button in adding storage:
~~~
2023-12-24 16:13:16,511 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 7-789818784650103765: Processing: { Ans: , MgmtId: 206863094307331, via: 7, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,511 DEBUG [c.c.a.m.AgentManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,512 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage pool33 to the host7].
2023-12-24 16:13:16,517 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to attach storage pool33 to the host7] and content [Unable to attach storage pool33 to the host7].
2023-12-24 16:13:16,518 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to establish a connection between Host {"id":7,"name":"kvm-192-168-13-13","type":"Routing","uuid":"ef3f53cb-14fc-4eb4-a8ea-23a4896e973f"} and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 33 due to com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,606 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) (logid:) Seq 10-4530902700111432125: Processing: { Ans: , MgmtId: 206863094307331, via: 10, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,606 DEBUG [c.c.a.m.AgentManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,607 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage pool33 to the host10].
2023-12-24 16:13:16,611 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to attach storage pool33 to the host10] and content [Unable to attach storage pool33 to the host10].
2023-12-24 16:13:16,612 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to establish a connection between Host {"id":10,"name":"kvm-192-168-13-14","type":"Routing","uuid":"9f9f655a-fef4-49d7-aafa-900cabd51315"} and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 33 due to com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,698 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 11-7044474242137589771: Processing: { Ans: , MgmtId: 206863094307331, via: 11, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,698 DEBUG [c.c.a.m.AgentManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,699 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage pool33 to the host11].
2023-12-24 16:13:16,703 WARN [c.c.a.AlertManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to attach storage pool33 to the host11] and content [Unable to attach storage pool33 to the host11].
2023-12-24 16:13:16,704 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to establish a connection between Host {"id":11,"name":"kvm-192-168-13-15","type":"Routing","uuid":"f6018446-c9dd-428e-95db-f5b397ce9e37"} and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 33 due to com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,704 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No host can access storage pool {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"} on cluster 1
2023-12-24 16:13:16,708 DEBUG [c.c.s.StorageManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to add data store: Failed to access storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to access storage pool
2023-12-24 16:13:16,710 DEBUG [c.c.s.StorageManagerImpl] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to clean up storage pool: null
2023-12-24 16:13:16,710 INFO [c.c.a.ApiServer] (qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to add data store: Failed to access storage pool
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to attach Ceph as primary storage [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
li-liwen commented on issue #8405:
URL: https://github.com/apache/cloudstack/issues/8405#issuecomment-1874044875
It turns out that I didn't to install the RBD driver for KVM.
I am able to resolve the problem by installing the driver:
~~~
sudo apt-get install libvirt-daemon-driver-storage-rbd
~~~
Closing this issue...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to attach Ceph as primary storage [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
li-liwen commented on issue #8405:
URL: https://github.com/apache/cloudstack/issues/8405#issuecomment-1868679915
Thanks for the quick reply! However, the management server actually do have connections to the ceph clusters. I have tried install the Ceph-common package, ceph.conf, and keyring on the management server as well, but still no progress. Here is the output from the management server diagnosing the connection (192.168.13.251 is one ceph monitor server):
~~~
username@cloudstack:~$ ping 192.168.13.251
PING 192.168.13.251 (192.168.13.251) 56(84) bytes of data.
64 bytes from 192.168.13.251: icmp_seq=1 ttl=63 time=0.255 ms
64 bytes from 192.168.13.251: icmp_seq=2 ttl=63 time=0.239 ms
^C
--- 192.168.13.251 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.239/0.247/0.255/0.008 ms
username@cloudstack:~$ telnet 192.168.13.251 6789
Trying 192.168.13.251...
Connected to 192.168.13.251.
Escape character is '^]'.
??H??v027???
^]
telnet> Connection closed.
username@cloudstack:~$ telnet 192.168.13.251 3300
Trying 192.168.13.251...
Connected to 192.168.13.251.
Escape character is '^]'.
ceph v2
^]
telnet> Connection closed.
sanmingyun@cloudstack:~$
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to attach Ceph as primary storage [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8405:
URL: https://github.com/apache/cloudstack/issues/8405#issuecomment-1868551151
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to attach Ceph as primary storage [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
li-liwen closed issue #8405: Unable to attach Ceph as primary storage
URL: https://github.com/apache/cloudstack/issues/8405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to attach Ceph as primary storage [cloudstack]
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #8405:
URL: https://github.com/apache/cloudstack/issues/8405#issuecomment-1868552701
Can your ACS mgmt servers ping your ceph monitors? If I recall correctly, when you first add a new ceph cluster, the ACS MGMT servers are involved directly, after that initial primary storage provisioning, the ACS agent on the KVM host creates all the images. So, if your managed servers can't reach your Ceph mons, you might need to temporarily route the networks to establish the new primary storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org