You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/09/15 08:02:26 UTC

[GitHub] [cloudstack] ravening opened a new issue #5452: Console proxy timeouts when there are multiple subnets

ravening opened a new issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452


   <!--
   Verify first that your issue/request is not already reported on GitHub.
   Also test if the latest release and main branch are affected too.
   Always add information AFTER of these HTML comments, but no need to delete the comments.
   -->
   
   ##### ISSUE TYPE
   <!-- Pick one below and delete the rest -->
    * Bug Report
    
   
   ##### COMPONENT NAME
   <!--
   Categorize the issue, e.g. API, VR, VPN, UI, etc.
   -->
   ~~~
   Console Proxy (NoVNC)
   ~~~
   
   ##### CLOUDSTACK VERSION
   <!--
   New line separated list of affected versions, commit ID for issues on main branch.
   -->
   
   ~~~
   4.14 onwards
   ~~~
   
   ##### CONFIGURATION
   <!--
   Information about the configuration if relevant, e.g. basic network, advanced networking, etc.  N/A otherwise
   -->
   Advanced networking
   Multiple subnets for management cidr (172.30.34.* and 172.30.36.*)
   
   ##### OS / ENVIRONMENT
   <!--
   Information about the environment if relevant, N/A otherwise
   -->
   Ubuntu 16 hosts
   
   ##### SUMMARY
   <!-- Explain the problem/feature briefly -->
   The console for a vm timeouts when there are multiple subnets configured.
   If we try to access the console of the vm which is running on the host with primary subnet then everything works fine.
   If we try to open the console of the vm which is running on the host with second subnet then the console hangs
   
   Looks like console proxy creates a static route for the second subnet which breaks the connection
   
   Moving the vm back to the hypervisor which has the primary subnet, fixes the issue.
   This issue was not present in cloudstack 4.7 version but the issues exists in 4.14 onwards. It probably might exist in 4.11 also
   
   ##### STEPS TO REPRODUCE
   <!--
   For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
   
   For new features, show how the feature would be used.
   -->
   
   <!-- Paste example playbooks or commands between quotes below -->
   ~~~
   1. Configure multiple subnets for the management cidr (172.30.34.* and 172.30.26.*)
   2. In our case few hypervisors had 172.30.34* subnet configured and few others had 172.30.36.*
   3. Here 172.30.34.* is the main subnet and 172.30.26.* is second subnet
   4. Deploy a VM on host with 172.30.34* and 172.30.36.* subnets
   5. open the console for the console proxy vm
   6. Now try to open the console for the vm running on host with subnet 172.30.34.* This works fine
   7. Now open the console for the vm running on the host with 172.30.36.* as the subnet
   8. The console hangs
   
   Below is the routing table
   
   # route -n
   Kernel IP routing table
   Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
   0.0.0.0         37.48.69.14     0.0.0.0         UG    0      0        0 eth2
   10.0.0.0        172.30.34.254   255.0.0.0       UG    0      0        0 eth1
   37.48.69.0      0.0.0.0         255.255.255.240 U     0      0        0 eth2
   169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
   172.16.0.0      172.30.34.254   255.240.0.0     UG    0      0        0 eth1
   172.30.34.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
   172.30.36.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
   172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   192.168.0.0     172.30.34.254   255.255.0.0     UG    0      0        0 eth1
   
   
   Below are the Ip address of the hosts where vm is running and these routes are created the moment we tried to access console of that vm. So these routes should not be created. When we delete this route manually, it works fine
   
   172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
   ~~~
   
   <!-- You can also paste gist.github.com links for larger files -->
   
   ##### EXPECTED RESULTS
   <!-- What did you expect to happen when running the steps above? -->
   
   ~~~
   The console for all vm's running on all hosts should work fine without any timeout
   ~~~
   
   ##### ACTUAL RESULTS
   <!-- What actually happened? -->
   
   <!-- Paste verbatim command output between quotes below -->
   ~~~
   The console for the vm's running on the hosts with secondary subnet timeouts/hangs
   ~~~
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919842524


   > @weizhouapache so the console works for initial 30 seconds and then freezes.
   > the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`
   
   @ravening 
   I think you could consult networking engineers how to solve this issue in switch configurations
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening closed issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening closed issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919796277


   @ravening is it related to your pr https://github.com/apache/cloudstack/pull/4678 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919800892


   > @ravening is it related to your pr #4678 ?
   
   @weizhouapache no... because that pr adds the cidr to the route but not the specific ip address.
   if you see the route -n output, we can see that only specific ip routes are added when we access the console
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-1029748065


   @nvazquez no.. I will close this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919823261


   @weizhouapache so the console works for initial 30 seconds and then freezes.
   the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-920637913


   > @weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also
   
   @ravening 
   the issue might exist with legacy vnc console and new novnc console introduced in 4.14
   
   ```
   ConsoleProxy.ensureRoute(uri.getHost());
   ```
   You did not see the issue because you did not use legacy vnc console but use a different novnc implementation.
   
   I suggest you to fix it by network configuration changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919844245


   @weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache closed issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
weizhouapache closed issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919839162


   > @weizhouapache so the console works for initial 30 seconds and then freezes.
   > the static route is added by python scripts or the java code? I added some logs under `CsRoute.py` `cs/CsStaticRoutes.py` but didnt see that in `/var/log/cloud.log`
   
   @ravening 
   it is added by java
   
   ```
       public void ensureRoute(String address) {
           if (_localgw != null) {
               if (s_logger.isDebugEnabled())
                   s_logger.debug("Ensure route for " + address + " via " + _localgw);
   
               // this method won't be called in high frequency, serialize access
               // to script execution
               synchronized (this) {
                   try {
                       addRouteToInternalIpOrCidr(_localgw, _eth1ip, _eth1mask, address);
                   } catch (Throwable e) {
                       s_logger.warn("Unexpected exception while adding internal route to " + address, e);
                   }
               }
           }
       }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] nvazquez commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-1029587392


   Hi @ravening is this issue still valid?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] ravening commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
ravening commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-919792998


   @weizhouapache @rhtyd have you guys faces this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] rhtyd commented on issue #5452: Console proxy timeouts when there are multiple subnets

Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #5452:
URL: https://github.com/apache/cloudstack/issues/5452#issuecomment-920597787


   Never seen/heard it @ravening


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org