You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "Shaofeng SHI (JIRA)" <ji...@apache.org> on 2019/04/01 03:13:00 UTC

[jira] [Commented] (KYLIN-3913) Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata

    [ https://issues.apache.org/jira/browse/KYLIN-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806353#comment-16806353 ] 

Shaofeng SHI commented on KYLIN-3913:
-------------------------------------

hi shaohui, thanks for the reporting; I checked the pr, it changes to query the job step each time, my concern is, will it cause too many requests (each request only fetch one job step output) to the metastore in a short time? If you have done some benchmark, please share the performance data you collected, thank you!

> Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
> -----------------------------------------------------------------------------
>
>                 Key: KYLIN-3913
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3913
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Liu Shaohui
>            Priority: Major
>
> In a big cluster, there will be many job info left in the metadata. The kylin server will be OOM when search the jobs with a long time range. The reason is that ExecutableManager will load all job output info into memory when search a job.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)