[jira] [Comment Edited] (HIVE-11777) implement an option to have single ETL strategy for multiple directories

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903760#comment-14903760 ] 

Sergey Shelukhin edited comment on HIVE-11777 at 9/23/15 1:16 AM:
------------------------------------------------------------------

[~gopalv] this is the method to not make metastore call per split. Thoughts? Another alternative I thought about was to make metastore call from one thread, allowing directory listings to accumulate until it completes, for the next call.


was (Author: sershe):
[~gopalv] this is the method to not make metastore call per split. Thoughts? 

> implement an option to have single ETL strategy for multiple directories
> ------------------------------------------------------------------------
>
>                 Key: HIVE-11777
>                 URL: https://issues.apache.org/jira/browse/HIVE-11777
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>             Fix For: hbase-metastore-branch
>
>         Attachments: HIVE-11777.patch
>
>
> In case of metastore footer PPD we don't want to call PPD call with all attendant SARG, MS and HBase overhead for each directory. If we wait for some time (10ms? some fraction of inputs?) we can do one call without losing overall perf. 
> For now make it time based.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)