You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/02/27 09:14:56 UTC
[GitHub] [incubator-shardingsphere] nevereverever opened a new issue #4499:
sharding-proxy CPU load too high
nevereverever opened a new issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499
sharding-proxy version:4.0.0
I ran my test program ,it's a simple insert,10 threads, 1000 insert per thread in 1 hour, my virtual machine CPU(4 cores) load more than 350%.Sharding rule is a simple hash.
top command output:
```bash
[app@csybapp3 conf]$ top
top - 17:05:30 up 10 days, 8:18, 1 user, load average: 10.70, 9.10, 5.01
Tasks: 273 total, 1 running, 272 sleeping, 0 stopped, 0 zombie
%Cpu0 : 96.3 us, 1.0 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 96.0 us, 1.3 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 95.7 us, 1.0 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu3 : 95.7 us, 1.3 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 8174068 total, 168588 free, 5522060 used, 2483420 buff/cache
KiB Swap: 8388604 total, 8386804 free, 1800 used. 1811484 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9768 app 20 0 8022356 1.1g 15356 S 369.1 14.3 36:01.51 java
```
I grab the program stack,find that CPU consumption here:
```java
"pool-17-thread-1" #53 prio=5 os_prio=0 tid=0x00007f4c28008000 nid=0x2ce9 runnable [0x00007f4c08b24000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaringClass0(Native Method)
at java.lang.Class.getDeclaringClass(Class.java:1235)
at java.lang.Class.getEnclosingClass(Class.java:1277)
at java.lang.Class.getSimpleBinaryName(Class.java:1443)
at java.lang.Class.getSimpleName(Class.java:1309)
at org.apache.shardingsphere.sql.parser.core.extractor.util.ExtractorUtils.isMatchedNode(ExtractorUtils.java:158)
at org.apache.shardingsphere.sql.parser.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:129)
at org.apache.shardingsphere.sql.parser.core.extractor.impl.dml.insert.InsertValuesExtractor.extractExpressionSegments(InsertValuesExtractor.java:59)
at org.apache.shardingsphere.sql.parser.core.extractor.impl.dml.insert.InsertValuesExtractor.extract(InsertValuesExtractor.java:53)
at org.apache.shardingsphere.sql.parser.core.extractor.SQLSegmentsExtractorEngine.extract(SQLSegmentsExtractorEngine.java:52)
at org.apache.shardingsphere.sql.parser.core.SQLParseKernel.parse(SQLParseKernel.java:59)
at org.apache.shardingsphere.sql.parser.SQLParseEngine.parse0(SQLParseEngine.java:70)
at org.apache.shardingsphere.sql.parser.SQLParseEngine.parse(SQLParseEngine.java:52)
at org.apache.shardingsphere.core.route.router.sharding.ShardingRouter.parse(ShardingRouter.java:86)
at org.apache.shardingsphere.core.route.StatementRoutingEngine.route(StatementRoutingEngine.java:53)
at org.apache.shardingsphere.core.SimpleQueryShardingEngine.route(SimpleQueryShardingEngine.java:58)
at org.apache.shardingsphere.core.BaseShardingEngine.executeRoute(BaseShardingEngine.java:85)
at org.apache.shardingsphere.core.BaseShardingEngine.shard(BaseShardingEngine.java:68)
at org.apache.shardingsphere.shardingproxy.backend.communication.jdbc.wrapper.StatementExecutorWrapper.doShardingRoute(StatementExecutorWrapper.java:79)
at org.apache.shardingsphere.shardingproxy.backend.communication.jdbc.wrapper.StatementExecutorWrapper.route(StatementExecutorWrapper.java:65)
at org.apache.shardingsphere.shardingproxy.backend.communication.jdbc.JDBCDatabaseCommunicationEngine.execute(JDBCDatabaseCommunicationEngine.java:83)
at org.apache.shardingsphere.shardingproxy.backend.text.query.QueryBackendHandler.execute(QueryBackendHandler.java:54)
at org.apache.shardingsphere.shardingproxy.frontend.mysql.command.query.text.query.MySQLComQueryPacketExecutor.execute(MySQLComQueryPacketExecutor.java:73)
at org.apache.shardingsphere.shardingproxy.frontend.command.CommandExecutorTask.executeCommand(CommandExecutorTask.java:92)
at org.apache.shardingsphere.shardingproxy.frontend.command.CommandExecutorTask.run(CommandExecutorTask.java:72)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] terrymanu closed issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
terrymanu closed issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] nevereverever commented on issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
nevereverever commented on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-592268760
Almost,I read the source code,these series of parse cause the CPU rise,especially here:
```java
public static Collection<ParserRuleContext> getAllDescendantNodes(final ParserRuleContext node, final RuleName ruleName) {
Collection<ParserRuleContext> result = new LinkedList<>();
for (ParserRuleContext each : getAllNodes(node)) {
if (isMatchedNode(each, ruleName)) {
result.add(each);
}
}
return result;
}
```
```java
private static Collection<ParserRuleContext> getAllNodes(final ParserRuleContext node) {
Collection<ParserRuleContext> result = new LinkedList<>();
LinkedList<ParserRuleContext> stack = new LinkedList<>();
stack.add(node);
while (!stack.isEmpty()) {
ParserRuleContext each = stack.pop();
result.add(each);
ParserRuleContext[] childNodes = getChildrenNodes(each).toArray(new ParserRuleContext[0]);
for (int i = childNodes.length - 1; i >= 0; i--) {
stack.push(childNodes[i]);
}
}
return result;
}
```
A large number of loops have been executed in this "while".
> @nevereverever
> You means "ExtractorUtils.isMatchedNode()" is called too many times, so raise CPU consumption?
@kimmking
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] nevereverever edited a comment on issue
#4499: sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
nevereverever edited a comment on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-592272502
ExtractorUtils.isMatchedNode() is called too many times,too.
And this's one reason.
```java
private static boolean isMatchedNode(final ParserRuleContext node, final RuleName ruleName) {
return ruleName.getName().equals(node.getClass().getSimpleName());
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking edited a comment on issue
#4499: sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
kimmking edited a comment on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-591892604
@nevereverever
You means "ExtractorUtils.isMatchedNode()" is called too many times, so raise CPU consumption?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking commented on issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
kimmking commented on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-596040094
The parser is the core of Sharding, and will consume the most CPU.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] terrymanu commented on issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-596180418
Please try to 5.x, the class ExtractorUtils had been removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking commented on issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
kimmking commented on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-591892604
You means "ExtractorUtils.isMatchedNode" method is called too many times, so raise CPU consumption ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] nevereverever commented on issue #4499:
sharding-proxy CPU load too high
Posted by GitBox <gi...@apache.org>.
nevereverever commented on issue #4499: sharding-proxy CPU load too high
URL: https://github.com/apache/incubator-shardingsphere/issues/4499#issuecomment-592272502
ExtractorUtils.isMatchedNode() is called too many times,too.
And this's one reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services