You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by akashrn5 <gi...@git.apache.org> on 2018/07/30 14:22:08 UTC

[GitHub] carbondata pull request #2583: [CARBONDATA-2803]fix wrong datasize calculati...

GitHub user akashrn5 opened a pull request:

    https://github.com/apache/carbondata/pull/2583

    [CARBONDATA-2803]fix wrong datasize calculation

    ### Problem: When a single block has multiple blocklets, then the data size written in table status file was wrong.
    
    ### Solution:
    The indexFileMap contains all the blocklets and index file mapping. For example, if one block contains 3 blocklets, then index file entry will list all the blocklets of all the block present in it. Since all the three blocklets will have the same block path, so just get the size of one block path for exact data size and avoid wrong datasize calculation.
    
    How this ### tested:
    tested with 1TB block, without fix it was giving three times size as data size, and after fix gives the correct data size.
    
    
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [x] Any interfaces changed?
     NA
     - [x] Any backward compatibility impacted?
     NA
     - [x] Document update required?
    NA
     - [x] Testing done
    Done in 3 node cluster
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. 
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/akashrn5/incubator-carbondata datasize

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2583
    
----
commit 30ba2f50cf68f4f0af1278b26c0408243919997e
Author: akashrn5 <ak...@...>
Date:   2018-07-30T14:11:34Z

    fix wrong datasize calculation

----


---

[GitHub] carbondata pull request #2583: [CARBONDATA-2803]fix wrong datasize calculati...

Posted by akashrn5 <gi...@git.apache.org>.

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2583#discussion_r207113233
  
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2651,8 +2651,17 @@ public static int isFilterPresent(byte[][] filterValues,
           carbonIndexSize = getCarbonIndexSize(fileStore, locationMap);
           for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
             // get the size of carbondata files
    +        String tempBlockFilePath = null;
             for (String blockFile : entry.getValue()) {
    -          carbonDataSize += FileFactory.getCarbonFile(blockFile).getSize();
    +          // the indexFileMap contains all the blocklets and index file mapping. For example, if one
    +          // block contains 3 blocklets, then entry.getValue() will list all the blocklets of all
    +          // the block present in it. Since all the three blocklets will have the same block path,
    +          // so just get the size of one block path for exact data size and avoid wrong datasize
    +          // calculation.
    +          if (!blockFile.equals(tempBlockFilePath)) {
    --- End diff --
    
    i will revert this change


---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation

Posted by brijoobopanna <gi...@git.apache.org>.

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    retest this please



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7655/



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation

Posted by ravipesala <gi...@git.apache.org>.

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6066/



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation and ...

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7726/



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7634/



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation and ...

Posted by brijoobopanna <gi...@git.apache.org>.

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    retest sdv please



---

[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation and ...

Posted by ravipesala <gi...@git.apache.org>.

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2583
  
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6118/



---