You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ZhouKang (Jira)" <ji...@apache.org> on 2019/09/18 10:40:00 UTC
[jira] [Comment Edited] (KYLIN-4169) too many logs while
DataModelManager init, cause the first RESTful API hang for a long time
[ https://issues.apache.org/jira/browse/KYLIN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932289#comment-16932289 ]
ZhouKang edited comment on KYLIN-4169 at 9/18/19 10:39 AM:
-----------------------------------------------------------
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at org.apache.kylin.metadata.model.DataModelManager.<init>(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List<String> paths = store.collectResourceRecursively(resRootPath, resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole project. At this moment, the remain models (which is valid actually) have not been loaded to the cache.
If there are 100 models in a project, the logs count will be 1+2+3+...+99
was (Author: zhoukangcn):
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at org.apache.kylin.metadata.model.DataModelManager.<init>(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List<String> paths = store.collectResourceRecursively(resRootPath, resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole project. At this moment, the remain models (which is valid actually) have not been loaded to the cache.
> too many logs while DataModelManager init, cause the first RESTful API hang for a long time
> -------------------------------------------------------------------------------------------
>
> Key: KYLIN-4169
> URL: https://issues.apache.org/jira/browse/KYLIN-4169
> Project: Kylin
> Issue Type: Bug
> Reporter: ZhouKang
> Priority: Major
>
> Too many logs while DataModelManager init. It is will happend after startup or reload metastore.
> In our production enviroment，there are 120,000+ lines.
> {code:java}
> // code placeholder
> grep 'is missing or unloaded yet' kylin.log.8|wc -l
> 121861
> {code}
>
> which cause the first sql query's time up to 40s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)