You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/12/15 10:45:08 UTC
[GitHub] [cloudstack] xuanyuanaosheng opened a new issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
xuanyuanaosheng opened a new issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776
##### ISSUE TYPE
* Bug Report
The related issue is : https://github.com/apache/cloudstack/issues/5741
I create a volumes using Ceph RBD Storage,
But Is is failed when I attach this volemes to a running VM.
---------------------
The related info is:
I using command in KVM node and find the volume has created in ceph pool.
```
# rbd -p cloudstack ls
4f558b9c-4850-4727-b454-d0bcc0773284
baff76c8-0d6e-446f-a2df-541d285b33f0
```
The cloudstack agent error logs is
```
2021-12-15 18:51:14,815 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:44e5aea5) Trying to fetch storage pool b90eae9d-973c-362c-8afc-af88f0743892 from libvirt
2021-12-15 18:51:14,815 WARN [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:44e5aea5) Caught:
com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'b90eae9d-973c-362c-8afc-af88f0743892'
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:505)
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:424)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.connectPhysicalDisk(KVMStoragePoolManager.java:129)
at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.attachVolume(KVMStorageProcessor.java:1332)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:158)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'b90eae9d-973c-362c-8afc-af88f0743892'
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.storagePoolLookupByUUIDString(Unknown Source)
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:433)
... 17 more
```
The cloudstack web UI error is
```
message.attach.volume.failed: whzpromalert001
Failed to attach volume whzpromalert001 to VM whzpromalert001 due to: class com.cloud.agent.api.Answer cannot be cast to class org.apache.cloudstack.storage.command.AttachAnswer (com.cloud.agent.api.Answer and org.apache.cloudstack.storage.command.AttachAnswer are in unnamed module of loader 'app')
```
Could you please give some advices ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-999241914
@wido
I use below command to handle this problem.
```
virsh pool-list
virsh pool-info b90eae9d-973c-362c-8afc-af88f0743892
virsh pool-undefine b90eae9d-973c-362c-8afc-af88f0743892
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng edited a comment on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng edited a comment on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-998465025
@wido I restart the VM，But It is also report the same errors.
Today I add another kvm host into the cluster. Now the cluster has two nodes.
- 10.26.246.5 (First added)
- 10.26.246.6 (Later added)
The additon info:
1. The ceph RBD primary storage scope is CLUSTER mode
The First added node report errors， But the Later added node works fine.
Now the first kvm node (10.26.246.5) has can not connect to the managerment node, the cloudstack agent error:
**Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6**
The details error as below:
```
2021-12-21 12:30:45,444 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:45,446 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:45,446 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:45,451 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:5e42d071) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:45,462 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:45,463 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
2021-12-21 12:30:45,463 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:ed0d935b) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) SSL: Handshake done
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connected to 10.25.2.173:8250
2021-12-21 12:30:45,560 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:ed0d935b) Failed to parse Script output: Host.OS.Version=
2021-12-21 12:30:45,595 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Attempting to create storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 (Filesystem) in libvirt
2021-12-21 12:30:45,595 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:ed0d935b) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-21 12:30:45,596 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Found existing defined storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1, using it.
2021-12-21 12:30:45,597 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Trying to fetch storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 from libvirt
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Proccess agent startup answer, agent id = 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Set agent id 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Startup Response Received: agent id = 0
2021-12-21 12:30:45,800 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Attempting to create storage pool 60b59087-7c53-3058-a50c-f50737e556bc (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Found existing defined storage pool 60b59087-7c53-3058-a50c-f50737e556bc, using it.
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Trying to fetch storage pool 60b59087-7c53-3058-a50c-f50737e556bc from libvirt
2021-12-21 12:30:45,873 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Attempting to create storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Found existing defined storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff, using it.
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Trying to fetch storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff from libvirt
2021-12-21 12:30:45,897 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:d1d208d9) Connected to the host: 10.25.2.173
2021-12-21 12:30:45,942 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Attempting to create storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Found existing defined storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa, using it.
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Trying to fetch storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa from libvirt
2021-12-21 12:30:46,053 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:46,056 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:46,056 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:46,060 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:2e37d7fd) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
```
I have searched this error on google, But I can not get any help. Any idears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng edited a comment on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng edited a comment on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-998465025
@wido I restart the VM，But It is also report the same errors. I do not know why this happen.
------
Today I add another kvm host into the cluster. Now the cluster has two nodes.
- 10.26.246.5 (First added)
- 10.26.246.6 (Later added)
The additon info:
1. The ceph RBD primary storage scope is CLUSTER mode
The First added node report errors， But the Later added node works fine.
Now the first kvm node (10.26.246.5) has can not connect to the managerment node, the cloudstack agent error:
**Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6**
The details error as below:
```
2021-12-21 12:30:45,444 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:45,446 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:45,446 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:45,451 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:5e42d071) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:45,462 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:45,463 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
2021-12-21 12:30:45,463 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:ed0d935b) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) SSL: Handshake done
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connected to 10.25.2.173:8250
2021-12-21 12:30:45,560 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:ed0d935b) Failed to parse Script output: Host.OS.Version=
2021-12-21 12:30:45,595 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Attempting to create storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 (Filesystem) in libvirt
2021-12-21 12:30:45,595 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:ed0d935b) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-21 12:30:45,596 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Found existing defined storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1, using it.
2021-12-21 12:30:45,597 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Trying to fetch storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 from libvirt
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Proccess agent startup answer, agent id = 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Set agent id 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Startup Response Received: agent id = 0
2021-12-21 12:30:45,800 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Attempting to create storage pool 60b59087-7c53-3058-a50c-f50737e556bc (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Found existing defined storage pool 60b59087-7c53-3058-a50c-f50737e556bc, using it.
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Trying to fetch storage pool 60b59087-7c53-3058-a50c-f50737e556bc from libvirt
2021-12-21 12:30:45,873 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Attempting to create storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Found existing defined storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff, using it.
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Trying to fetch storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff from libvirt
2021-12-21 12:30:45,897 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:d1d208d9) Connected to the host: 10.25.2.173
2021-12-21 12:30:45,942 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Attempting to create storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Found existing defined storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa, using it.
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Trying to fetch storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa from libvirt
2021-12-21 12:30:46,053 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:46,056 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:46,056 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:46,060 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:2e37d7fd) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
```
I have searched this error on google, But I can not get any help. Any idears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] wido commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
wido commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-997877664
Have you tried to stop/start the VM? Maybe hotplug doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-998465025
@wido I restart the VM，But It is also report the same errors.
I add another kvm host into the cluster. Now the cluster has two nodes.
- 10.26.246.5 (First added)
- 10.26.246.6 (Later added)
The additon info:
1. The ceph RBD primary storage scope is CLUSTER mode
The First added node report errors， But the Later added node works fine.
Now the first kvm node (10.26.246.5) has can not connect to the managerment node, the cloudstack agent error:
**Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6**
The details error as below:
```
2021-12-21 12:30:45,444 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:45,446 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:45,446 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:45,451 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:5e42d071) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:45,462 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:45,463 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
2021-12-21 12:30:45,463 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:ed0d935b) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) SSL: Handshake done
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connected to 10.25.2.173:8250
2021-12-21 12:30:45,560 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:ed0d935b) Failed to parse Script output: Host.OS.Version=
2021-12-21 12:30:45,595 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Attempting to create storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 (Filesystem) in libvirt
2021-12-21 12:30:45,595 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:ed0d935b) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-21 12:30:45,596 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Found existing defined storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1, using it.
2021-12-21 12:30:45,597 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Trying to fetch storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 from libvirt
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Proccess agent startup answer, agent id = 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Set agent id 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Startup Response Received: agent id = 0
2021-12-21 12:30:45,800 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Attempting to create storage pool 60b59087-7c53-3058-a50c-f50737e556bc (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Found existing defined storage pool 60b59087-7c53-3058-a50c-f50737e556bc, using it.
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Trying to fetch storage pool 60b59087-7c53-3058-a50c-f50737e556bc from libvirt
2021-12-21 12:30:45,873 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Attempting to create storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Found existing defined storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff, using it.
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Trying to fetch storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff from libvirt
2021-12-21 12:30:45,897 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:d1d208d9) Connected to the host: 10.25.2.173
2021-12-21 12:30:45,942 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Attempting to create storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Found existing defined storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa, using it.
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Trying to fetch storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa from libvirt
2021-12-21 12:30:46,053 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:46,056 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:46,056 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:46,060 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:2e37d7fd) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
```
I have searched this error on google, But I can not get any help. Any idears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-1064724773
Great, thanks @xuanyuanaosheng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-1064718773
@nvazquez This issue has been resolved using the method above. I think this is a bug fomr ceph: https://github.com/rook/rook/issues/7746#issuecomment-825919401
The resolved problem is :
```
# ceph config get mon auth_expose_insecure_global_id_reclaim
false
```
I will close this issue. Thanks all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-997544111
@sureshanaparti Any idears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-1029554588
Hi @xuanyuanaosheng is this issue still valid or you have found a workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng closed issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng closed issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-997610367
> @sureshanaparti Any idears?
No idea. @wido @NuxRo @weizhouapache Any thoughts / comments on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng edited a comment on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng edited a comment on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-995422371
@weizhouapache @wido Could you please give some advice?
I use the disk.xml
```
<disk type='network' device='disk'>
<source protocol='rbd' name='cloudstack/baff76c8-0d6e-446f-a2df-541d285b33f0'>
<host name='10.29.44.1' port='6789'/>
</source>
<target dev='vdb' bus='virtio'/>
</disk>
```
And I attack this to i-2-263-VM using:
```
# virsh attach-device i-2-263-VM disk.xml
Device attached successfully
```
But I can not find vdb in i-2-263-VM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng commented on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng commented on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-995422371
@weizhouapache @wido
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] xuanyuanaosheng edited a comment on issue #5776: Ceph RBD Storage: Failed to attach volume XXXXXXX to VM XXXXXXXXXX
Posted by GitBox <gi...@apache.org>.
xuanyuanaosheng edited a comment on issue #5776:
URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-998465025
@wido I restart the VM，But It is also report the same errors. I do not know why this happen.
------
Today I add another kvm host into the cluster. Now the cluster has two nodes.
- 10.26.246.5 (First added)
- 10.26.246.6 (Later added)
The additon info:
1. The ceph RBD primary storage scope is CLUSTER mode
The First added node report errors， But the Later added node works fine.
Now the first kvm node (10.26.246.5) has can not connect to the managerment node, the cloudstack agent error:
**Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6**
The details error as below:
```
2021-12-21 12:30:45,444 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:45,446 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:45,446 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:45,451 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:5e42d071) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:45,462 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:45,463 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
2021-12-21 12:30:45,463 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:ed0d935b) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) SSL: Handshake done
2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connected to 10.25.2.173:8250
2021-12-21 12:30:45,560 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:ed0d935b) Failed to parse Script output: Host.OS.Version=
2021-12-21 12:30:45,595 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Attempting to create storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 (Filesystem) in libvirt
2021-12-21 12:30:45,595 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:ed0d935b) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-21 12:30:45,596 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Found existing defined storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1, using it.
2021-12-21 12:30:45,597 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Trying to fetch storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 from libvirt
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Proccess agent startup answer, agent id = 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Set agent id 0
2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Startup Response Received: agent id = 0
2021-12-21 12:30:45,800 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Attempting to create storage pool 60b59087-7c53-3058-a50c-f50737e556bc (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Found existing defined storage pool 60b59087-7c53-3058-a50c-f50737e556bc, using it.
2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Trying to fetch storage pool 60b59087-7c53-3058-a50c-f50737e556bc from libvirt
2021-12-21 12:30:45,873 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Attempting to create storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Found existing defined storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff, using it.
2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Trying to fetch storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff from libvirt
2021-12-21 12:30:45,897 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:d1d208d9) Connected to the host: 10.25.2.173
2021-12-21 12:30:45,942 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Attempting to create storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa (NetworkFilesystem) in libvirt
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Found existing defined storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa, using it.
2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Trying to fetch storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa from libvirt
2021-12-21 12:30:46,053 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt
2021-12-21 12:30:46,056 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it.
2021-12-21 12:30:46,056 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6
2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create the RBD storage pool, cleaning up the libvirt secret
2021-12-21 12:30:46,060 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:2e37d7fd) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress.
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed
2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173
2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250
```
I have searched this error on google, But I can not get any help. Any idears?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org