You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/08/17 06:10:45 UTC
[GitHub] [incubator-kyuubi] timothy65535 opened a new issue #940: Kyuubiserver consumes 600% cpu
timothy65535 opened a new issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940
### 1. Describe the bug
After several benchmarks in the offline environment, kyuubi consumes a lot of CPU.
```
[bigdata]$ jps
3188 Main
6036 Jps
4189 KyuubiServer
[bigdata]$ top
top - 09:34:21 up 23 min, 1 user, load average: 10.67, 7.85, 4.98
Tasks: 320 total, 1 running, 319 sleeping, 0 stopped, 0 zombie
%Cpu(s): 48.2 us, 23.9 sy, 0.0 ni, 27.4 id, 0.0 wa, 0.5 hi, 0.0 si, 0.0 st
MiB Mem : 15806.0 total, 4738.4 free, 6085.0 used, 4982.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9306.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4189 bigdata 20 0 10.1g 311272 22484 S 633.3 1.9 48:12.21 java
```
### 2. Environments
#### 2.1 Versions
- Kyuubi
- [x] 1.3.0
- Spark
- [x] 3.1.x
#### 2.2 Kyuubi Configurations
#### 2.3 Spark Configurations
### 3. Steps to reproduce
When more clients connect to kyuubi server with different user, kyuubiserver consumes more and more cpu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900027434
> Can you help to see which thread use high cpu ratio ? Use `top -Hp 4189 ` and use `jstack` to find that thread.
👍 Yeah, I'm looking into the `process-logger-capture` thread.
```
Threads: 53 total, 6 running, 47 sleeping, 0 stopped, 0 zombie
%Cpu(s): 31.9 us, 22.8 sy, 0.0 ni, 44.8 id, 0.0 wa, 0.4 hi, 0.0 si, 0.0 st
MiB Mem : 15806.0 total, 4770.5 free, 6052.9 used, 4982.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9338.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5222 bigdata 20 0 10.1g 311272 22484 R 99.3 1.9 10:00.91 process-logger-
5044 bigdata 20 0 10.1g 311272 22484 R 99.0 1.9 10:03.98 process-logger-
4730 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:13.24 process-logger-
4911 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:06.42 process-logger-
4523 bigdata 20 0 10.1g 311272 22484 R 98.0 1.9 10:41.55 process-logger-
5769 bigdata 20 0 10.1g 311272 22484 R 96.7 1.9 2:46.80 process-logger-
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 edited a comment on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
timothy65535 edited a comment on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900045139
> Seems we can sleep a while if read line is null
For `BufferedReader`, we can use `reader.ready()`.
```
while (true) {
if (reader.ready()) {
var line: String = reader.readLine
// logic
} else {
Thread.sleep(300)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900029481
Here has a `while(true)`
```
val redirect: Runnable = { () =>
try {
val maxErrorSize = conf.get(KyuubiConf.ENGINE_ERROR_MAX_SIZE)
var line: String = reader.readLine
while (true) {
......
}
} catch {
case _: IOException =>
case _: InterruptedException =>
} finally {
reader.close()
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900026429
Can you help to see which thread use high cpu ratio ? Use `top -Hp 4189 ` and use `jstack` to find that thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900045139
> Seems we can sleep a while if read line is null
For `BufferedReader`, we can use `reader.ready()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
ulysses-you closed issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900035652
Seems we can sleep a while if read line is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 edited a comment on issue #940: Kyuubiserver consumes 600% cpu
Posted by GitBox <gi...@apache.org>.
timothy65535 edited a comment on issue #940:
URL: https://github.com/apache/incubator-kyuubi/issues/940#issuecomment-900027434
> Can you help to see which thread use high cpu ratio ? Use `top -Hp 4189 ` and use `jstack` to find that thread.
Yeah, I'm looking into the `process-logger-capture` thread.
```
Threads: 53 total, 6 running, 47 sleeping, 0 stopped, 0 zombie
%Cpu(s): 31.9 us, 22.8 sy, 0.0 ni, 44.8 id, 0.0 wa, 0.4 hi, 0.0 si, 0.0 st
MiB Mem : 15806.0 total, 4770.5 free, 6052.9 used, 4982.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9338.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5222 bigdata 20 0 10.1g 311272 22484 R 99.3 1.9 10:00.91 process-logger-
5044 bigdata 20 0 10.1g 311272 22484 R 99.0 1.9 10:03.98 process-logger-
4730 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:13.24 process-logger-
4911 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:06.42 process-logger-
4523 bigdata 20 0 10.1g 311272 22484 R 98.0 1.9 10:41.55 process-logger-
5769 bigdata 20 0 10.1g 311272 22484 R 96.7 1.9 2:46.80 process-logger-
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org