You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "sbrueseke (via GitHub)" <gi...@apache.org> on 2025/03/06 14:55:14 UTC
[D] console not working after system vm has been recreated on a different cluster [cloudstack]
GitHub user sbrueseke created a discussion: console not working after system vm has been recreated on a different cluster
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Console Proxy
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.19.1.2
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Management server on Ubuntu 22.04 LTS
KVM hypervisor on Ubuntu 22.04 LTS
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
We run a multiple cluster installation. After deleting console proxy vm, Cloudstack recreated a new system vm on antoher cluster.
After that console access for instances running on a different cluster than the console proxy vm is running on, did not work anymore. But after stopping and starting an instance, console access was working again.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
1. detroy console proxy system vm
2. let CS recreate the console proxy system vm and hope it will be created on another cluster
3. try to open a console on an instance running on a different cluster as the console proxy is running on (not working)
4. stop and start this instance and try again (working again)
5. try to open the console on an instance running on the same cluster as the console proxy is running on (working)
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
console should work on all instances even after CS recreated it on another cluster
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
console is only working on instances running on the same cluster as the console proxy system vm. After stop and start of an instance, console is working again.
~~~
GitHub link: https://github.com/apache/cloudstack/discussions/10521
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@weizhouapache no firewall on hosts
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415177
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@sbrueseke , I converted this to an q&a item. Any updates?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415190
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke closed a discussion: console not working after system vm has been recreated on a different cluster
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Console Proxy
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.19.1.2
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Management server on Ubuntu 22.04 LTS
KVM hypervisor on Ubuntu 22.04 LTS
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
We run a multiple cluster installation. After deleting console proxy vm, Cloudstack recreated a new system vm on antoher cluster.
After that console access for instances running on a different cluster than the console proxy vm is running on, did not work anymore. But after stopping and starting an instance, console access was working again.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
1. detroy console proxy system vm
2. let CS recreate the console proxy system vm and hope it will be created on another cluster
3. try to open a console on an instance running on a different cluster as the console proxy is running on (not working)
4. stop and start this instance and try again (working again)
5. try to open the console on an instance running on the same cluster as the console proxy is running on (working)
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
console should work on all instances even after CS recreated it on another cluster
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
console is only working on instances running on the same cluster as the console proxy system vm. After stop and start of an instance, console is working again.
~~~
GitHub link: https://github.com/apache/cloudstack/discussions/10521
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
> i-21-283-VM was created in June this year and last updated on July 16th. Are you able to reproduce this issue on your installation? Everything worked fine, before we recreated the console proxy VM and it was put on another cluster.
@sbrueseke
no, everything works in my testing
can you check if the hosts/vms are time-synced ?
you can enable debugging in CPVM and retry. You might get more information in /var/log/cloud.log
```
sed -i "s/INFO/DEBUG/g" /usr/local/cloud/systemvm/conf/log4j-cloud.xml
systemctl restart cloud
```
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415183
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
> root@pc-kvm-1:~$ virsh domdisplay i-21-283-VM vnc://10.10.1.21:1
>
> root@v-529-VM:~# ping -c1 10.10.1.21 PING 10.10.1.21 (10.10.1.21): 56 data bytes 64 bytes from 10.10.1.21: icmp_seq=0 ttl=64 time=0.291 ms --- 10.10.1.21 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.291/0.291/0.291/0.000 ms
>
> root@v-529-VM:~# telnet 10.10.1.21 5901 Trying 10.10.1.21... Connected to 10.10.1.21. Escape character is '^]'. RFB 003.008
@sbrueseke
Has the vm been running for long time, let's say more than 1 year ?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415180
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
> @weizhouapache no firewall on hosts
can you also check these steps ?
- on the kvm host (another cluster), run "virsh domdisplay <vm name>" or "virsh vncdisplay <vm name>"
```
root@kvm1:~# virsh domdisplay i-2-1192-VM
vnc://10.0.xxx.xxx:2
```
- in CPVM, try ping and telnet
```
root@v-296-VM:~# ping -c1 10.0.xxx.xxx
# if it does not work, can you share output of `ip route` in the CPVM ?
root@v-296-VM:~# telnet 10.0.xxx.xxx 5902
Trying 10.0.xxx.xxx...
Connected to 10.0.xxx.xxx.
Escape character is '^]'.
RFB 003.008
```
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415178
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke added a comment to the discussion: console not working after system vm has been recreated on a different cluster
Clusters are in the same pod.
Cluster A
root@pc-kvm-1:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.21
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.21
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
Cluster B
root@pc-kvm-6:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.26
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.26
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415175
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
> Clusters are in the same pod.
>
> Cluster A root@pc-kvm-1:~# ip route default via 10.10.1.1 dev cloudbr0 proto static 10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.21 10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.21 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
>
> Cluster B root@pc-kvm-6:~# ip route default via 10.10.1.1 dev cloudbr0 proto static 10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.26 10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.26 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
@sbrueseke
looks like simple setup which should work.
any firewall rules on the kvm hosts ?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415176
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke added a comment to the discussion: console not working after system vm has been recreated on a different cluster
root@pc-kvm-1:~$ virsh domdisplay i-21-283-VM
vnc://10.10.1.21:1
root@v-529-VM:~# ping -c1 10.10.1.21
PING 10.10.1.21 (10.10.1.21): 56 data bytes
64 bytes from 10.10.1.21: icmp_seq=0 ttl=64 time=0.291 ms
--- 10.10.1.21 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.291/0.291/0.291/0.000 ms
root@v-529-VM:~# telnet 10.10.1.21 5901
Trying 10.10.1.21...
Connected to 10.10.1.21.
Escape character is '^]'.
RFB 003.008
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415179
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@weizhouapache are you able to reproduce this issue?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415182
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland closed the discussion with a comment: console not working after system vm has been recreated on a different cluster
closed this for lack of interaction, please reopen or open a new item if applicable.
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-13002547
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
GitHub user sbrueseke added a comment to the discussion: console not working after system vm has been recreated on a different cluster
i-21-283-VM was created in June this year and last updated on July 16th. Are you able to reproduce this issue on your installation? Everything worked fine, before we recreated the console proxy VM and it was put on another cluster.
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415181
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
GitHub user rohityadavcloud added a comment to the discussion: console not working after system vm has been recreated on a different cluster
cc @nvazquez @weizhouapache @DaanHoogland to help review/triage if it's a regression or some kind of limitation we should address in next 4.19.2/4.20.1
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415172
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@sbrueseke
have you fixed the issue ?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12562351
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@sbrueseke are the clusters in the same pod? (I don't think it relevant, but it might)
is there a recent update involved that changed behaviour, that you know off?
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415174
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] console not working after system vm has been recreated on a different cluster [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: console not working after system vm has been recreated on a different cluster
@sbrueseke
can you share some network information of the clusters ?
if possible, please share the output of "ip route" in the CPVM as well
GitHub link: https://github.com/apache/cloudstack/discussions/10521#discussioncomment-12415173
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org