You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "fengYu (JIRA)" <ji...@apache.org> on 2017/10/11 11:10:00 UTC

[jira] [Updated] (KYLIN-2929) speed up Dump file performance

     [ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

fengYu updated KYLIN-2929:
--------------------------
    Labels: Performance  (was: )

> speed up Dump file performance
> ------------------------------
>
>                 Key: KYLIN-2929
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2929
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: fengYu
>            Assignee: fengYu
>              Labels: Performance
>
> when I work on KYLIN-2926, I find coprocessor will dump to disk once estimatedMemSize is bigger than spillThreshold, and found that spill data size is extraordinary smaller that estimatedMemSize, in my case dump file size is about 8MB and spillThreshold is setting to 3GB.   
> So, I try to keep the spill data in memory rather than write the file to disk immediately, and when those in-memory spill data reach the threshold, write all spill files together.
> In my case, the coprocessor process cost time drop from 22s to 16s, it is about 30% upgrade。



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)