[jira] [Commented] (KYLIN-2192) More Robust Global Dictionary

["Dayue Gao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735518#comment-15735518 ] 

Dayue Gao commented on KYLIN-2192:
----------------------------------

Found a bug in AppendTrieDictionary, which cause Kylin to load all data of AppendTrieDictionary into memory when it's about to add values to it. Fixed in https://github.com/apache/kylin/commit/fbb7ed921a8b63c3b62cb85bf64fb79ba650431d 

> More Robust Global Dictionary
> -----------------------------
>
>                 Key: KYLIN-2192
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2192
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>    Affects Versions: v1.5.4.1
>            Reporter: Yerui Sun
>            Assignee: Yerui Sun
>             Fix For: v1.6.1
>
>         Attachments: KYLIN-2192.2.patch
>
>
> Global dictionary have been released over 2 months, I've received some feedbacks and bug reports. Here's the patch to make global dictionary more robust, including some functional improvements.
> * Break through 255 bytes limitation for value, but still recommend value length less than 8K, avoiding stack overflow error;
> * Fix 'Value not exists' or stack overflow error when dict size is larger than 1GB, the root cause is similar with KYLIN-1834; A check tool also provided for check corrupted or not of existing dict data;
> * Support parallel dictionary building in one job server, used for parallel segments building;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)