You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "tuanhoangth1603 (via GitHub)" <gi...@apache.org> on 2023/03/13 13:32:58 UTC
[GitHub] [cloudstack] tuanhoangth1603 opened a new issue, #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
tuanhoangth1603 opened a new issue, #7336:
URL: https://github.com/apache/cloudstack/issues/7336
Hello,
When integrating with CEPH RBD, because multiple monitor hosts are supported in version 4.18 as the issue https://github.com/apache/cloudstack/pull/6792.
So on Cloudstack 4.17, I configured a HAProxy node to load balance the CEPH monitor service endpoint, but Cloudstack cannot connect when adding a new primary storage. Cloudstack agent host raises an error in agent.log:
"Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor at MON_IP:6789: No such file or directory."
Here is my HaProxy configuration:
`
frontend ceph-mon
bind *:6789
mode tcp
default_backend ceph-mon-backend
backend ceph-mon-backend
mode tcp
balance roundrobin
server ceph-mon-1 MON_IP:6789 check
frontend ceph-mon33
bind *:6789
mode tcp
default_backend ceph-mon-backend33
backend ceph-mon-backend33
mode tcp
balance roundrobin
server ceph-mon-1 MON_IP:3300 check
`
Thanks for your help !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466201568
I have updated the previous commennt. In summary:
Management -> ceph-mon: OK
Haproxy -> ceph-mon: OK
Management -> Haproxy -> ceph-mon: Not working and repeats a log line that I gave in the above comment
And I have setup with admin permission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466339563
@weizhouapache Based on your experience, please guide me more if you have faced this error. I tried the rDNS that you mentioned. I can create primary storage through a domain using r-DNS. However, when creating an instance, the agent reports an error:
```
2023-03-13 22:00:31,821 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:19c6ecd0) Failed to convert from /mnt/fc83a653-108a-3e03-92cf-11ef4af18ad7/e11436365ee-3ead-8a7f-368ca26c12bd.qcow2 to rbd:cloudstack/a431408a-13a7-4a6b-8823-b8be106c913e:mon_host=ceph-mon.cs.local:auth_supported=cephx:id=cloudstack:key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx==:rbd_default_format=2:client_mount_timeout=30 the error was: qemu-img: rbd:cloudstack/a431408a-13a7-4a6b-8823-b8be106c913e:mon_host=ceph-mon.cs.local:auth_pported=cephx:id=cloudstack:key=xxxxxxxxxxxxxxxxxxx==:rbd_default_format=2:client_mount_timeout=30: error while converting raw: error rbd create: File exists
```
The management and host can resolve and ping to ceph-mon.cs.local successfully.
Thanks so much !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] kiwiflyer commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466191700
What permissions do you have setup on the user on the Ceph side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] kiwiflyer commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466163815
@tuanhoangth1603 Can the management server(s) talk to haproxy?
If I recall correctly, the management server attempts a connection before adding the RBD pool. All subsequent calls to Ceph are made from the CloudStack host agents.
Most installations I've seen use round-robin DNS for the Ceph monitors. The new 4.18 that is in RC status supports specifying multiple management servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] kiwiflyer commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466477046
@tuanhoangth1603
Run virsh pool-list on the host and look to see whether the RBD pool is listed.
If not, try restarting libvirtd and then restart the cloudstack-agent.
CloudStack will attempt to build the pool definition in libvirt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466404966
@kiwiflyer @weizhouapache I tried, but still not work
I deleted all image, delete ceph pool then make a new one and successfully add this pool for primary storage. But some errors still happened when I am creating an instance:
```
2023-03-13 22:45:19,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Didn't find an existing storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 by UUID, checking for pools with duplicate paths
2023-03-13 22:45:29,059 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:0b1fc6b1) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2023-03-13 22:45:29,284 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:f55f79f8) Fetching health check result for 169.254.40.66 and executing fresh checks: false
2023-03-13 22:45:42,361 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-13 22:45:42,380 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-13 22:45:42,394 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-13 22:45:49,413 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack-cfox' exist?: No such file or directory
2023-03-13 22:45:49,413 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Failed to create the RBD storage pool, cleaning up the libvirt secret
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 closed issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 closed issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
URL: https://github.com/apache/cloudstack/issues/7336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] kiwiflyer commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466352389
@tuanhoangth1603 CloudStack creates "cached" versions of your templates in Ceph to make the provisioning faster. It looks like a previous attempt at adding storage may have partially worked, and one of the templates had already been copied over. CloudStack is trying to do it again and is complaining that the image already exists.
Fix this by deleting any images you see in the RBD pool, delete the pool and re-add it, then attempt to add the RBD pool to CloudStack again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1467264467
Thanks @kiwiflyer but not changed.
This is my procedure:
1. I removed the old CEPH primary storage that was added with one IP of ceph-mon, which was properly working. The result of the virsh pool-list command is:
```
Name State Autostart
------------------------------------------------------------
12d49b2f-792c-3aca-b27e-f9b9d5e858b2 active no
78ee67e3-53f0-4a62-b583-3f56975bd8a3 active no
```
12d49b2f-792c-3aca-b27e-f9b9d5e858b2: UUID of NFS Primary storage
78ee67e3-53f0-4a62-b583-3f56975bd8a3: UUID of local storage pool
2. I deleted the CEPH primary storage and CEPH pool, and then re-created a new CEPH pool. I added that pool to Cloudstack primary storage again via the rDNS of ceph-mon that points to 3 IPs, including an IP that was added previously. After adding the new pool, the result of the virsh list command is:
```
Name State Autostart
------------------------------------------------------------
12d49b2f-792c-3aca-b27e-f9b9d5e858b2 active no
78ee67e3-53f0-4a62-b583-3f56975bd8a3 active no
a1ace81c-e92d-31fb-b0bd-d8ac41219a78 active no
```
a1ace81c-e92d-31fb-b0bd-d8ac41219a78: this is new UUID of CEPH pool
3. I create a new instance with CEPH primary storage, it's failed and show the error log as below:
```
2023-03-14 09:15:28,998 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:8378662e) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2023-03-14 09:15:29,254 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-4:null) (logid:a6654827) Fetching health check result for 169.254.40.66 and executing fresh checks: false
2023-03-14 09:15:42,361 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:c2df74eb) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:15:42,377 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:c2df74eb) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:15:42,389 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:c2df74eb) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:16:14,605 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:5f00792d) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:16:14,630 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:5f00792d) Asking libvirt to refresh storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2
2023-03-14 09:16:14,642 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Trying to fetch storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 from libvirt
2023-03-14 09:16:14,649 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Asking libvirt to refresh storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78
2023-03-14 09:16:44,668 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Attempting to create storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 (RBD) in libvirt
2023-03-14 09:16:44,690 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 was not found running in libvirt. Need to create it.
2023-03-14 09:16:44,690 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Didn't find an existing storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 by UUID, checking for pools with duplicate paths
2023-03-14 09:16:44,741 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:5f00792d) Trying to fetch storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 from libvirt
2023-03-14 09:17:44,858 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:95c62b13) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:17:44,882 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:95c62b13) Asking libvirt to refresh storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2
2023-03-14 09:17:44,937 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Trying to fetch storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 from libvirt
2023-03-14 09:17:44,944 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Asking libvirt to refresh storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78
2023-03-14 09:18:14,961 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Attempting to create storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 (RBD) in libvirt
2023-03-14 09:18:14,981 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 was not found running in libvirt. Need to create it.
2023-03-14 09:18:14,982 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Didn't find an existing storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 by UUID, checking for pools with duplicate paths
2023-03-14 09:18:45,010 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack-cfox' exist?: No such file or directory
2023-03-14 09:18:45,010 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:95c62b13) Failed to create the RBD storage pool, cleaning up the libvirt secret
2023-03-14 09:19:45,132 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:079ae294) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
2023-03-14 09:19:45,158 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:079ae294) Asking libvirt to refresh storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2
2023-03-14 09:19:45,213 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:079ae294) Trying to fetch storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 from libvirt
2023-03-14 09:19:45,216 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:079ae294) Attempting to create storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 (RBD) in libvirt
2023-03-14 09:19:45,218 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:079ae294) Storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 was not found running in libvirt. Need to create it.
2023-03-14 09:19:45,218 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:079ae294) Didn't find an existing storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 by UUID, checking for pools with duplicate paths
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1556862974
@tuanhoangth1603
is this issue solved in your environment ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466425211
> @weizhouapache I think we probably could if we were able to checksum the image. I've had an issue before where the qemuimg copy failed and the RBD image was corrupt.
>
@kiwiflyer yes, this is also what I was worries about.
> The entire process is a little bit too "fire and forget" at the moment and needs some checking to be added.
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466188493
@kiwiflyer From management server, I can telnet to HaProxy on port 3300 and 6789.
HaProxy node also connect to monitor host via command:
```
ceph mon -c /etc/ceph/ceph.conf --mon-host MON_IP:6789 stat
```
But on management server, it return a continuously repeated log line
```
2023-03-13T20:47:58.199+0700 7f286effd700 0 --2- MON_IP:0/3789996788 >> v2:MON_IP:3300/0 conn(0x7f28701009a0 0x7f2870102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1467531144
> a1ace81c-e92d-31fb-b0bd-d8ac41219a78
@tuanhoangth1603
it looks like timeout while refresh the pool.
can you execute "time virsh pool-refresh a1ace81c-e92d-31fb-b0bd-d8ac41219a78" on the kvm host
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466238567
@tuanhoangth1603
This might be caused by haproxy configurations.
There is another way, please refer to
https://www.42on.com/redundant-ceph-monitors-with-round-robin-dns/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466395214
> @tuanhoangth1603 CloudStack creates "cached" versions of your templates in Ceph to make the provisioning faster. It looks like a previous attempt at adding storage may have partially worked, and one of the templates had already been copied over. CloudStack is trying to do it again and is complaining that the image already exists.
>
> Fix this by deleting any images you see in the RBD pool, delete the pool and re-add it, then attempt to add the RBD pool to CloudStack again.
agree with @kiwiflyer
You can retry after removing the images from ceph rbd.
@kiwiflyer
do you think if we can ignore the error "error while converting raw: error rbd create: File exists" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466432118
> @kiwiflyer @weizhouapache I tried, but still not work I deleted all image, delete ceph pool then make a new one and successfully add this pool for primary storage. But some errors still happened when I am creating an instance:
>
> ```
> 2023-03-13 22:45:19,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Didn't find an existing storage pool a1ace81c-e92d-31fb-b0bd-d8ac41219a78 by UUID, checking for pools with duplicate paths
> 2023-03-13 22:45:29,059 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:0b1fc6b1) Expected 1 answers while executing SetMonitorServiceCommand but received 3
> 2023-03-13 22:45:29,284 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:f55f79f8) Fetching health check result for 169.254.40.66 and executing fresh checks: false
> 2023-03-13 22:45:42,361 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
> 2023-03-13 22:45:42,380 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
> 2023-03-13 22:45:42,394 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ffff363e) Trying to fetch storage pool 12d49b2f-792c-3aca-b27e-f9b9d5e858b2 from libvirt
> 2023-03-13 22:45:49,413 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack-cfox' exist?: No such file or directory
> 2023-03-13 22:45:49,413 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6758e709) Failed to create the RBD storage pool, cleaning up the libvirt secret
> ```
>
> I'd like to reinstall agent host however I think we can keep this to track an issue more and avoid in the future!
@tuanhoangth1603
it is not needed to reinstall agent/host.
could you please give more details what you did ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] kiwiflyer commented on issue #7336: HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
kiwiflyer commented on issue #7336:
URL: https://github.com/apache/cloudstack/issues/7336#issuecomment-1466405771
@weizhouapache I think we probably could if we were able to checksum the image. I've had an issue before where the qemuimg copy failed and the RBD image was corrupt.
The entire process is a little bit too "fire and forget" at the moment and needs some checking to be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org