You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "Harif-Rahman (via GitHub)" <gi...@apache.org> on 2023/04/12 05:28:41 UTC
[GitHub] [shardingsphere] Harif-Rahman opened a new issue, #25136: Heap memory increase due to sql dynamic comment
Harif-Rahman opened a new issue, #25136:
URL: https://github.com/apache/shardingsphere/issues/25136
## Bug Report
**For English only**, other languages will not accept.
Spring boot version : 2.1.5.RELEASE
JDK : amazoncorretto:8u332
ShardingSphere version : 5.2.0
We have introduced a change to pass the request id for every outgoing sql for tracing purpose.
After this change we have observed increase in jvm heap memory
<img width="1620" alt="image" src="https://user-images.githubusercontent.com/48013254/231359262-43e71c98-61ea-46df-b52f-0ffd7528d88d.png">
<img width="1628" alt="image" src="https://user-images.githubusercontent.com/48013254/231359312-2e6d36e7-20e7-4d53-885d-51e4f9a06fa4.png">
Statement inspector class :
public class SqlStatementInspector implements StatementInspector {
private static final int REQUEST_PATH_MAX_LENGTH = 50;
@OverRide
public String inspect(String sql) {
// add request id in sql as comment eg : *select * from table t1 / R:1 23 P: /api/v1 */ **
return sql + /* R: getrequestid() P: getrequestPath() */
}
}
Before report a bug, make sure you have:
- Searched open and closed [GitHub issues](https://github.com/apache/shardingsphere/issues).
- Read documentation: [ShardingSphere Doc](https://shardingsphere.apache.org/document/current/en/overview).
Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot reproduce it on current information, we will **close it**.
Please answer these questions before submitting your issue. Thanks!
### Which version of ShardingSphere did you use?
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
### Expected behavior
### Actual behavior
### Reason analyze (If you can)
### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
### Example codes for reproduce this issue (such as a github link).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] RaigorJiang commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1710047482
Closed due to no response.
If this problem persists, please reopen it or submit a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] Harif-Rahman commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "Harif-Rahman (via GitHub)" <gi...@apache.org>.
Harif-Rahman commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1506953663
No it wasn't and also suspecting the increase in sql parser cache since the trace id is unique every time it was not reusing the cache results.
i was also referring to this PR : https://github.com/apache/shardingsphere/pull/22901 as it also has a similar use case.
Meanwhile please let me know if anything needed from me for debugging this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] RaigorJiang commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1506918722
Hi @Harif-Rahman
Was there a similar increase in memory before adding the comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] RaigorJiang closed issue #25136: Heap memory increase due to sql dynamic comment
Posted by "RaigorJiang (via GitHub)" <gi...@apache.org>.
RaigorJiang closed issue #25136: Heap memory increase due to sql dynamic comment
URL: https://github.com/apache/shardingsphere/issues/25136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] Harif-Rahman commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "Harif-Rahman (via GitHub)" <gi...@apache.org>.
Harif-Rahman commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1510134703
@tuichenchuxin
i used sqlCommentParseEnabled as false and this is my configuration for that
CacheOption parseTreeCacheOption = new CacheOption(128, 1024L);
CacheOption sqlStatementCacheOption = new CacheOption(2000, 65535L);
Collection<RuleConfiguration> ruleConfigurations = new ArrayList<>();
ruleConfigurations.add(shardingRuleConfig);
ruleConfigurations.add(new SQLParserRuleConfiguration(false, parseTreeCacheOption, sqlStatementCacheOption));
But the sql query going to mysql is going without comments itself but ideally we need to ignore only on parse and we need to pass the actual sql with comments to mysql right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "tuichenchuxin (via GitHub)" <gi...@apache.org>.
tuichenchuxin commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1507853800
I agree with your suspect. The cache can't be used cause of different comments. Consider changing the configuration of the SQL parser rule to reduce the cache size.
In addition, I have another idea, if `sqlCommentParseEnabled: false`, you can remove the comment information from the cached SQL of sqlPaserEngine, so that you can use the same cache even if you comment different SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #25136: Heap memory increase due to sql dynamic comment
Posted by "tuichenchuxin (via GitHub)" <gi...@apache.org>.
tuichenchuxin commented on issue #25136:
URL: https://github.com/apache/shardingsphere/issues/25136#issuecomment-1537694083
If it is false, then shardingsphere will not care about the content of comments other than /*SHARDINGSPHERE_HINT*/. You can check whether the SQL entered into the shardingsphere has comment information. If you use mysql-client, you need to add `-c` parameter to pass comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org