You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2017/06/22 05:20:35 UTC
Re: Apache Kylin performance problem
Hi Zhifeng,
Thanks for the sharing. A few questions for it: Is this change made in
Apache Calcite (and which version) ? Will it bring any concurrency issue?
If it is safe, that should be merged to Calcite. Thx!
2017-06-22 10:48 GMT+08:00 苏 志锋 <ro...@live.com>:
> Environment
>
> Apache kylin 1.6.0
>
> Jdk1.8.0_72 64-Bit
>
> any one cube and any one SQL
>
>
>
> Problem
>
> When the concurrency users count increase, many request’s response
> time become longer and longer, but some are normal.
>
> This is because when calcite parse the sql.
>
> The function getBindable(org.apache.calcite.adapter.enumerable.EnumerableInterpretable
> ,line 119) dynamic create a bindable instance. The bindalbe instance call
> loadClass(java.lang.ClassLoader line 401) function which has a
> synchronized java code many times. So, one thead may wait for another.
>
> After I change the following code, the performance become much better.
>
> 1,org.codehaus.janino.ByteArrayClassLoader
>
> +static {
>
> + ClassLoader.registerAsParallelCapable();
>
> + }
>
> 2,org.apache.calcite.adapter.enumerable.EnumerableRelImplementor
>
> line 159 return Expressions.classDecl(Modifier.PUBLIC,
>
> -line 160 "Baz",
>
> +line 160 "Baz"+System.currentTimeMillis(),
>
> line 161 null,
>
> line 162 Collections.<Type>singletonList(Bindable.class),
>
> line 163 memberDeclarations);
>
--
Best regards,
Shaofeng Shi 史少锋
答复: Apache Kylin performance problem
Posted by 苏 志锋 <ro...@live.com>.
The first change is in janino-2.7.6.jar(group org.codehaus.janino). The second change is in calcite-core-1.11.0.jar. After performance test we have not found any concurrency issue. In fact, the change had not solve the problem finally, we are looking for another better solution.
The follow is three lines of a query log. The first line and the second line has 32s waits.
2017-06-07 16:02:04,968 INFO [Query cb5c8e7d-c1c0-411a-a463-347c23ea0486-7624] routing.QueryRouter:68 : The realizations remaining: [SalesProjectCube(CUBE)] And the final chosen one is the first one
2017-06-07 16:02:36,533 DEBUG [Query cb5c8e7d-c1c0-411a-a463-347c23ea0486-7624] enumerator.OLAPEnumerator:109 : query storage...
2017-06-07 16:02:36,551 INFO [Query cb5c8e7d-c1c0-411a-a463-347c23ea0486-7624] gtrecord.GTCubeStorageQueryBase:109 : Cuboid identified: cube=SalesProjectCube, cuboidId=1, groupsD=[], otherDimsD=[]
--------------------------------------
中国建设银行厦门开发中心
苏志锋
________________________________
发件人: ShaoFeng Shi <sh...@apache.org>
发送时间: 2017年6月22日 13:20:35
收件人: user; dev
主题: Re: Apache Kylin performance problem
Hi Zhifeng,
Thanks for the sharing. A few questions for it: Is this change made in Apache Calcite (and which version) ? Will it bring any concurrency issue? If it is safe, that should be merged to Calcite. Thx!
2017-06-22 10:48 GMT+08:00 苏 志锋 <ro...@live.com>>:
Environment
Apache kylin 1.6.0
Jdk1.8.0_72 64-Bit
any one cube and any one SQL
Problem
When the concurrency users count increase, many request’s response time become longer and longer, but some are normal.
This is because when calcite parse the sql.
The function getBindable(org.apache.calcite.adapter.enumerable.EnumerableInterpretable ,line 119) dynamic create a bindable instance. The bindalbe instance call loadClass(java.lang.ClassLoader line 401) function which has a synchronized java code many times. So, one thead may wait for another.
After I change the following code, the performance become much better.
1,org.codehaus.janino.ByteArrayClassLoader
+static {
+ ClassLoader.registerAsParallelCapable();
+ }
2,org.apache.calcite.adapter.enumerable.EnumerableRelImplementor
line 159 return Expressions.classDecl(Modifier.PUBLIC,
-line 160 "Baz",
+line 160 "Baz"+System.currentTimeMillis(),
line 161 null,
line 162 Collections.<Type>singletonList(Bindable.class),
line 163 memberDeclarations);
--
Best regards,
Shaofeng Shi 史少锋