You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "wi3dzma (via GitHub)" <gi...@apache.org> on 2026/08/27 10:22:22 UTC
[I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
wi3dzma opened a new issue, #13989:
URL: https://github.com/apache/cloudstack/issues/13989
### problem
CloudStack builds its Ceph RBD connection strings with the legacy Ceph option `auth_supported=cephx.` This option was removed in Ceph Tentacle 20.2.4. As a result, every RBD operation performed by the KVM agent through librados now fails, and the agent cannot bring up the RBD storage pool:
```
**agent.err:**
libvirt: Storage Driver error : internal error: failed to set RADOS option: auth_supported
**agent.log:**
2026-08-27 00:00:13,468 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:e0478a61) Failed to create RBD storage pool: org.libvirt.LibvirtException: internal error: failed to set RADOS option: auth_supported
2026-08-27 00:00:13,468 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:e0478a61) Failed to create the RBD storage pool, cleaning up the libvirt secret
```
In ACS Host state is **Disconnected**
Code:
_plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMPhysicalDisk.java_, lines 56-62
### versions
ACS: 4.22.1.1
Hypervisor OS: KVM on Rocky Linux 9.8
libvirt: 11.10.0
Ceph client packages: librados2 / librbd1 / ceph-common 2:20.2.4-0.el9
Ceph cluster: 20.2.4 Tentacle
### The steps to reproduce the bug
1. Deploy a KVM host with Ceph RBD primary storage and cephx enabled.
2. Upgrade the Ceph client packages on the KVM host to 20.2.4 (Tentacle): librados2, librbd1, ceph-common.
3. Restart libvirtd and cloudstack-agent.
### What to do about it?
Remove `auth_supported`, it is redundant. When `id` and `key` are supplied, the client negotiates cephx anyway, and when they are not, it falls back to whatever `auth_client_required` in `ceph.conf` specifies.
It is also possible to use `auth_client_required` instead of `auth_supported`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5437796754
thanks @wi3dzma
added to 4.22.2 milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-6165596620
> gitlab: https://gitlab.com/libvirt/libvirt/-/work_items/918
> proposed fix: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/3TVM5DVH3MOIQAM6ZUS5TYBLNSA6UA7K/
the gitlab issue has been fixed in libvirt: https://gitlab.com/libvirt/libvirt/-/commit/3af8cc51d5f596866cf971b750a0546c0205cf02
For users who encounter the issue, please use Ceph 20.2.2 or 19.2.5 (not the latest versions) for now, and wait the OS vendors to backport the fix and create new libvirt releases.
closing this issue as it is not in the scope of CloudStack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5582120452
> I ran into this same issue rebuilding my lab. For now we're just holding our KVM hosts on Ceph 20.2.2. Anyone building a fresh CS install with Ceph Tentacle (latest) is going to hit this issue.
>
> RE: PR [#13991](https://github.com/apache/cloudstack/pull/13991) - while it is required, I don't think the PR fixes this on its own. The PR fixes the qemu-img operations (template copy to primary, volume copies, snapshot backup) so it is necessary, but there's also the error in the log above coming from libvirt. That libvirt log is where libvirt's RBD storage backend sets `auth_supported` itself when it opens the connection for the pool, so the connection to the storage pool will still fail on a 20.2.4 client with the PR applied. As far as I can see libvirt hasn't fixed this yet and Ceph hasn't reverted the removal.
>
> Proxmox ran into the same thing (https://bugzilla.proxmox.com/show_bug.cgi?id=7961) and ended up putting the option back in their own Ceph build rather than waiting on upstream.
>
> So [#13991](https://github.com/apache/cloudstack/pull/13991) is still needed for the qemu-img side, but this issue probably needs to stay open until libvirt catches up. Not sure if there's anything else that should be done communications wise - as there will be deployments failing if people are just using cepf-tentacle (latest).
@MitchDrage agreed on all counts — I hit the exact same thing while triaging this for a user, and it lines up with your read.
Confirmed by reproducing directly with `virsh pool-create` (bypassing CloudStack entirely) against a Ceph Tentacle 20.2.4 cluster: even with #13991 applied, `LibvirtStorageAdaptor.createRBDStoragePool()` fails identically, because the failing `auth_supported` call lives entirely inside libvirt's own C code (src/storage/storage_backend_rbd.c, `virStorageBackendRBDOpenRADOSConn`) — it's hardcoded there and treated as fatal, with no CloudStack-side string or config reaching it.
[#13991](https://github.com/apache/cloudstack/issues/13991) is still correct/needed for the qemu-img-driven paths (template seeding, volume/snapshot copies), but the storage-pool-creation path is a separate libvirt bug, not something CloudStack code can route around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "akoskuczi-bw (via GitHub)" <gi...@apache.org>.
akoskuczi-bw commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5940205575
Workaround until the libvirtd fix is available
**Save this code as rados_compat.c**
```
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <dlfcn.h>
#include <errno.h>
typedef int (*rados_conf_set_fn)(
void *cluster,
const char *option,
const char *value
);
int rados_conf_set(void *cluster, const char *option, const char *value)
{
static rados_conf_set_fn real_rados_conf_set = NULL;
if (option && strcmp(option, "auth_supported") == 0) {
fprintf(stderr,
"[rados-compat] ignoring deprecated option: "
"auth_supported=%s\n",
value ? value : "(null)");
return 0;
}
if (!real_rados_conf_set) {
dlerror();
real_rados_conf_set =
(rados_conf_set_fn)dlsym(RTLD_NEXT, "rados_conf_set");
const char *err = dlerror();
if (err || !real_rados_conf_set) {
fprintf(stderr,
"[rados-compat] dlsym(rados_conf_set) failed: %s\n",
err ? err : "unknown error");
return -EIO;
}
}
return real_rados_conf_set(cluster, option, value);
}
```
**Then compile and install it:**
_gcc -shared -fPIC -O2 -Wall -Wextra -o librados_compat.so rados_compat.c -ldl
install -o root -g root -m 0755 librados_compat.so /usr/local/lib/librados_compat.so_
**Preload it into libvirtd:**
_systemctl edit libvirtd_
```
[Service]
Environment="LD_PRELOAD=/usr/local/lib/librados_compat.so"
```
_systemctl daemon-reload
systemctl restart libvirtd
systemctl restart cloudstack-agent_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "mathwest (via GitHub)" <gi...@apache.org>.
mathwest commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5940410020
Heya! I can confirm that this also hits hard on cloudstack 4.23 with ceph 19.2.6 client libraries. I just went through this and had to downgrade the libraries due to the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "MitchDrage (via GitHub)" <gi...@apache.org>.
MitchDrage commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5539456290
I ran into this same issue rebuilding my lab. For now we're just holding our KVM hosts on Ceph 20.2.2.
Anyone building a fresh CS install with Ceph Tentacle (latest) is going to hit this issue.
RE: PR #13991 - while it is required, I don't think the PR fixes this on its own.
The PR fixes the qemu-img operations (template copy to primary, volume copies, snapshot backup) so it is necessary, but there's also the error in the log above coming from libvirt.
That libvirt log is where libvirt's RBD storage backend sets `auth_supported` itself when it opens the connection for the pool, so the connection to the storage pool will still fail on a 20.2.4 client with the PR applied. As far as I can see libvirt hasn't fixed this yet and Ceph hasn't reverted the removal.
Proxmox ran into the same thing (https://bugzilla.proxmox.com/show_bug.cgi?id=7961) and ended up putting the option back in their own Ceph build rather than waiting on upstream.
So #13991 is still needed for the qemu-img side, but this issue probably needs to stay open until libvirt catches up.
Not sure if there's anything else that should be done communications wise - as there will be deployments failing if people are just using cepf-tentacle (latest).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache closed issue #13989: KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4
URL: https://github.com/apache/cloudstack/issues/13989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-5946703727
> Workaround until the libvirtd fix is available
>
> **Save this code as rados_compat.c**
>
> ```
> #define _GNU_SOURCE
>
> #include <stdio.h>
> #include <string.h>
> #include <dlfcn.h>
> #include <errno.h>
>
> typedef int (*rados_conf_set_fn)(
> void *cluster,
> const char *option,
> const char *value
> );
>
> int rados_conf_set(void *cluster, const char *option, const char *value)
> {
> static rados_conf_set_fn real_rados_conf_set = NULL;
>
> if (option && strcmp(option, "auth_supported") == 0) {
> fprintf(stderr,
> "[rados-compat] ignoring deprecated option: "
> "auth_supported=%s\n",
> value ? value : "(null)");
>
> return 0;
> }
>
> if (!real_rados_conf_set) {
> dlerror();
>
> real_rados_conf_set =
> (rados_conf_set_fn)dlsym(RTLD_NEXT, "rados_conf_set");
>
> const char *err = dlerror();
>
> if (err || !real_rados_conf_set) {
> fprintf(stderr,
> "[rados-compat] dlsym(rados_conf_set) failed: %s\n",
> err ? err : "unknown error");
>
> return -EIO;
> }
> }
>
> return real_rados_conf_set(cluster, option, value);
> }
> ```
>
> **Then compile and install it:** _gcc -shared -fPIC -O2 -Wall -Wextra -o librados_compat.so rados_compat.c -ldl install -o root -g root -m 0755 librados_compat.so /usr/local/lib/librados_compat.so_
>
> **Preload it into libvirtd:** _systemctl edit libvirtd_
>
> ```
> [Service]
> Environment="LD_PRELOAD=/usr/local/lib/librados_compat.so"
> ```
>
> _systemctl daemon-reload systemctl restart libvirtd systemctl restart cloudstack-agent_
@akoskuczi-bw
I applied a similar change several days ago, and the libvirt pool can be created successfully. However, VMs cannot be deployed due to a segmentation fault in the QEMU process.
As a workaround, I had to override the QEMU command and unset LD_PRELOAD before running the actual QEMU process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13989:
URL: https://github.com/apache/cloudstack/issues/13989#issuecomment-6015685524
gitlab: https://gitlab.com/libvirt/libvirt/-/work_items/918
proposed fix: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/3TVM5DVH3MOIQAM6ZUS5TYBLNSA6UA7K/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org