You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by "suo tong (JIRA)" <ji...@apache.org> on 2016/12/14 06:59:59 UTC

[jira] [Updated] (CARBONDATA-429) Remove unnecessary file name check in dictionary cache

     [ https://issues.apache.org/jira/browse/CARBONDATA-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

suo tong updated CARBONDATA-429:
--------------------------------
    Description: 
In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions.
And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get to check if the local is valid or not.  The local dictionary cache may be invalid after another job finished load data. 

  was:In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions.


> Remove unnecessary file name check in dictionary cache
> ------------------------------------------------------
>
>                 Key: CARBONDATA-429
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-429
>             Project: CarbonData
>          Issue Type: Sub-task
>          Components: core
>    Affects Versions: 0.1.1-incubating
>            Reporter: Jacky Li
>            Assignee: Jacky Li
>             Fix For: 1.0.0-incubating
>
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions.
> And in checkAndLoadDictionaryData, we get meta file's mtime from hdfs each time we call cache.get to check if the local is valid or not.  The local dictionary cache may be invalid after another job finished load data. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)