You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/09/15 08:02:26 UTC
[GitHub] [cloudstack] ravening opened a new issue #5452: Console proxy timeouts when there are multiple subnets
ravening opened a new issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Console Proxy (NoVNC)
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.14 onwards
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Advanced networking
Multiple subnets for management cidr (172.30.34.* and 172.30.36.*)
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
Ubuntu 16 hosts
##### SUMMARY
<!-- Explain the problem/feature briefly -->
The console for a vm timeouts when there are multiple subnets configured.
If we try to access the console of the vm which is running on the host with primary subnet then everything works fine.
If we try to open the console of the vm which is running on the host with second subnet then the console hangs
Looks like console proxy creates a static route for the second subnet which breaks the connection
Moving the vm back to the hypervisor which has the primary subnet, fixes the issue.
This issue was not present in cloudstack 4.7 version but the issues exists in 4.14 onwards. It probably might exist in 4.11 also
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
1. Configure multiple subnets for the management cidr (172.30.34.* and 172.30.26.*)
2. In our case few hypervisors had 172.30.34* subnet configured and few others had 172.30.36.*
3. Here 172.30.34.* is the main subnet and 172.30.26.* is second subnet
4. Deploy a VM on host with 172.30.34* and 172.30.36.* subnets
5. open the console for the console proxy vm
6. Now try to open the console for the vm running on host with subnet 172.30.34.* This works fine
7. Now open the console for the vm running on the host with 172.30.36.* as the subnet
8. The console hangs
Below is the routing table
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 37.48.69.14 0.0.0.0 UG 0 0 0 eth2
10.0.0.0 172.30.34.254 255.0.0.0 UG 0 0 0 eth1
37.48.69.0 0.0.0.0 255.255.255.240 U 0 0 0 eth2
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
172.16.0.0 172.30.34.254 255.240.0.0 UG 0 0 0 eth1
172.30.34.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
172.30.36.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
172.30.36.3 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.25 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.37 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.42 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.56 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
192.168.0.0 172.30.34.254 255.255.0.0 UG 0 0 0 eth1
Below are the Ip address of the hosts where vm is running and these routes are created the moment we tried to access console of that vm. So these routes should not be created. When we delete this route manually, it works fine
172.30.36.3 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.25 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.37 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.42 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
172.30.36.56 172.30.34.254 255.255.255.255 UGH 0 0 0 eth1
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
The console for all vm's running on all hosts should work fine without any timeout
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
The console for the vm's running on the hosts with secondary subnet timeouts/hangs
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919842524
> @weizhouapache so the console works for initial 30 seconds and then freezes.
> the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`
@ravening
I think you could consult networking engineers how to solve this issue in switch configurations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening closed issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening closed issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919796277
@ravening is it related to your pr https://github.com/apache/cloudstack/pull/4678 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919800892
> @ravening is it related to your pr #4678 ?
@weizhouapache no... because that pr adds the cidr to the route but not the specific ip address.
if you see the route -n output, we can see that only specific ip routes are added when we access the console
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-1029748065
@nvazquez no.. I will close this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919823261
@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-920637913
> @weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also
@ravening
the issue might exist with legacy vnc console and new novnc console introduced in 4.14
```
ConsoleProxy.ensureRoute(uri.getHost());
```
You did not see the issue because you did not use legacy vnc console but use a different novnc implementation.
I suggest you to fix it by network configuration changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919844245
@weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache closed issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
weizhouapache closed issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919839162
> @weizhouapache so the console works for initial 30 seconds and then freezes.
> the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`
@ravening
it is added by java
```
public void ensureRoute(String address) {
if (_localgw != null) {
if (s_logger.isDebugEnabled())
s_logger.debug("Ensure route for " + address + " via " + _localgw);
// this method won't be called in high frequency, serialize access
// to script execution
synchronized (this) {
try {
addRouteToInternalIpOrCidr(_localgw, _eth1ip, _eth1mask, address);
} catch (Throwable e) {
s_logger.warn("Unexpected exception while adding internal route to " + address, e);
}
}
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-1029587392
Hi @ravening is this issue still valid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919792998
@weizhouapache @rhtyd have you guys faces this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #5452: Console proxy timeouts when there are multiple subnets
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-920597787
Never seen/heard it @ravening
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org