You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "kohrar (via GitHub)" <gi...@apache.org> on 2024/02/21 18:58:52 UTC

[I] High CPU when using secure VNC / NIOConnection [cloudstack]

kohrar opened a new issue, #8692:
URL: https://github.com/apache/cloudstack/issues/8692

   
   ##### ISSUE TYPE
    * Bug Report
   
   ##### COMPONENT NAME
   ~~~
   Console Proxy
   ~~~
   
   ##### CLOUDSTACK VERSION
   ~~~
   4.18, 4.19
   ~~~
   
   ##### CONFIGURATION
   Clean install of Apache CloudStack 4.18 and 4.19
   
   ##### OS / ENVIRONMENT
   Rocky Linux 8.9
   
   ##### SUMMARY
   When using the secure KVM VNC console proxy in 4.18 or 4.19, the Console Proxy SystemVM uses 100% of its single CPU when handling the connection.
   
   A quick analysis of the java process suggests the high CPU usage is from the [NIOConnection](https://github.com/apache/cloudstack/blob/main/utils/src/main/java/com/cloud/utils/nio/NioConnection.java) components:
   ![Screenshot from 2024-02-21 11-48-20](https://github.com/apache/cloudstack/assets/4450716/f1f2867a-fe88-4ada-ae8c-a69797896f86)
   [NoVncClient getNextBytes](https://github.com/apache/cloudstack/blob/main/services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVncClient.java#L125):
   ![Screenshot from 2024-02-21 11-48-35](https://github.com/apache/cloudstack/assets/4450716/b1d09320-c105-4458-a836-7c2b5076cf05)
   
   The ~100% CPU usage is happening even when the VNC screen is blank (eg. when the screen is off).
   
   In the mean time, I've changed the console proxy's SystemVM compute offering with additional CPUs to reduce CPU contention as we've had some users complain VNC connections were not responding.
   
   If this is expected behavior of NIOConnection, it may be reasonable to increase the CPU allocations in the default console proxy compute offering.
   
   ##### STEPS TO REPRODUCE
   1. Connect to a KVM VNC session on ACS >4.18.
   2. Connect to the console proxy and watch the java process's CPU usage.
   
   ##### EXPECTED RESULTS
   Reasonable CPU usage when proxying a connection when little data (or screen changes) are happening.
   
   ##### ACTUAL RESULTS
   I am seeing ~100% CPU usage per active VNC connection even when nothing is happening on the session (eg. a black screen).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1965867719

   Fixed in https://github.com/apache/cloudstack/pull/8694


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1962589613

   Good work @kohrar , thanks. There will be a new system VM template that will require a bigger offering in the next major version. Setting the default will be bigger for sure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud closed issue #8692: Console Proxy VM: High CPU when using secure VNC / NIOConnection
URL: https://github.com/apache/cloudstack/issues/8692


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "kohrar (via GitHub)" <gi...@apache.org>.
kohrar commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1965860495

   Awesome. Thanks for your work on this, Daan!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "kohrar (via GitHub)" <gi...@apache.org>.
kohrar commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1958031001

   Looks like it's similar. The [`while(connectionAlive){}` in ConsoleProxyNoVncClient.java for the NIO case](https://github.com/apache/cloudstack/blob/main/services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVncClient.java#L123) likely just needs to sleep to stop being a tight loop.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1958001069

   Is this similar as the issue fixed by #6970?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "kohrar (via GitHub)" <gi...@apache.org>.
kohrar commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1962027968

   I've rebuilt the cloud-systemvm files and copied it into the systemVM. I think your change helps as the process is no longer pegging the CPU at 100%. Now it's down to ~25% for a single connection.
   
   ```
   # top -b -n 1 | head -n 15
   top - 21:34:44 up 31 days, 15 min,  2 users,  load average: 0.14, 0.18, 0.23
   Tasks:  78 total,   2 running,  75 sleeping,   1 stopped,   0 zombie
   %Cpu(s): 17.6 us, 11.8 sy,  0.0 ni, 70.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
   MiB Mem :    976.6 total,     94.3 free,    487.6 used,    394.7 buff/cache
   MiB Swap:    243.0 total,    242.7 free,      0.3 used.    405.0 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     92659 root      20   0 2880800 140376  30428 S  25.0  14.0   0:39.87 java
         1 root      20   0  101588  11248   7168 S   0.0   1.1   1:00.53 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:01.18 kthreadd
         3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
         4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
         6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
         8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
         9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tas+
   
   # nproc
   1
   
   ```
   
   If this is the expected CPU usage going forward, would it make sense to have another PR that changes the default compute offerings so the console proxy VM has more than 1 CPU by default?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1961798839

   I was thinking you should build yourself if you can. If you can't we have to think of something else. I can't give you access directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "kohrar (via GitHub)" <gi...@apache.org>.
kohrar commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1961672520

   @DaanHoogland Could you point me to how I can test out the fix in the Console Proxy VM?  Is there a way to get access to the built files by blue orangutan in that particular PR? Thanks for the quick PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Console Proxy VM: High CPU when using secure VNC / NIOConnection [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #8692:
URL: https://github.com/apache/cloudstack/issues/8692#issuecomment-1960938462

   @kohrar can you test #8694?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org