[jira] [Commented] (CARBONDATA-80) Dictionary values should be equally distributed in buckets while loading in memory

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401914#comment-15401914 ] 

ASF GitHub Bot commented on CARBONDATA-80:
------------------------------------------

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/44#discussion_r72965345
  
    --- Diff: core/src/main/java/org/carbondata/core/cache/dictionary/AbstractColumnDictionaryInfo.java ---
    @@ -241,22 +257,21 @@ private void decrementAccessCount() {
        */
       protected byte[] getDictionaryBytesFromSurrogate(int surrogateKey) {
         byte[] dictionaryValueInBytes = null;
    -    int totalSizeOfDictionaryChunksTraversed = 0;
    -    for (List<byte[]> oneDictionaryChunk : dictionaryChunks) {
    -      totalSizeOfDictionaryChunksTraversed =
    -          totalSizeOfDictionaryChunksTraversed + oneDictionaryChunk.size();
    -      // skip the dictionary chunk till surrogate key is lesser than size of
    -      // dictionary chunks traversed
    -      if (totalSizeOfDictionaryChunksTraversed < surrogateKey) {
    -        continue;
    +    if (surrogateKey > 0) {
    --- End diff --
    
    why we need this check ? As data loading surrogate key starts from 1 , so in any case if surrogate key is less than 1 then  there is some bug in unpacking the mdkey 


> Dictionary values should be equally distributed in buckets while loading in memory
> ----------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-80
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-80
>             Project: CarbonData
>          Issue Type: Improvement
>            Reporter: Manish Gupta
>            Assignee: Manish Gupta
>            Priority: Minor
>
> Whenever a query is executed, dictionary for columns queried is loaded in memory. For incremental loads dictionary values are loaded incrementally and thus one list contains several sub lists with dictionary values.
> The dictionary values on incremental load may not be equally distributed in the sub buckets and this might increase the search time of a value if there are too many incremental loads.
> Therefore the dictionary values should be divided equally in the sub buckets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)