[jira] [Commented] (DRILL-3918) Avoid extra loading of the metadata cache file

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952326#comment-14952326 ] 

ASF GitHub Bot commented on DRILL-3918:
---------------------------------------

GitHub user amansinha100 opened a pull request:

    https://github.com/apache/drill/pull/196

    DRILL-3918: During expansion save the metadata for future use.

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/amansinha100/incubator-drill refresh_metadata_bugs_1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #196
    
----
commit 2f261d97f4f9eb8f13479a82b1e39028484fc49a
Author: Aman Sinha <as...@maprtech.com>
Date:   2015-10-11T15:02:30Z

    DRILL-3918: During expansion save the metadata for future use.

----


> Avoid extra loading of the metadata cache file
> ----------------------------------------------
>
>                 Key: DRILL-3918
>                 URL: https://issues.apache.org/jira/browse/DRILL-3918
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>            Reporter: Aman Sinha
>
> The metadata cache file is currently being deserialized and read twice: once during {{ParquetFormatPlugin.expandSelection()}} that happens as part of the creation of DynamicDrillTable and once during ParquetGroupScan.  This was also pointed out by [~sphillips] in DRILL-3901.   We should avoid doing the read twice.  
> The performance issue is getting exposed more now because of the fix for DRILL-3917 which fixed the behavior of expandSelection() by reading the metadata cache file through the correct interface (it was previously erroring out and not spending any time in the expansion). This fix is needed for correct functionality.   However, performance numbers show a slowdown of about 2.7x for the 400K files test using caching.  In my view, this performance comparison is not very meaningful because of the prior bug.  
> This JIRA is to specifically targeting the extra load of the metadata cache file.  There are other opportunities for improvement (for instance reading from the metadata cache is single threaded whereas reading from parquet files gets parallelized.  That should be a separate JIRA).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)