[jira] [Resolved] (IMPALA-4847) Simplify the code for file/block metadata loading by manually calling listLocatedStatus() for each partition.

["bharath v (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

bharath v resolved IMPALA-4847.
-------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.11.0

https://gerrit.cloudera.org/#/c/7652/

> Simplify the code for file/block metadata loading by manually calling listLocatedStatus() for each partition.
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-4847
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4847
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>    Affects Versions: Impala 2.8.0
>            Reporter: Alexander Behm
>            Assignee: bharath v
>            Priority: Critical
>             Fix For: Impala 2.11.0
>
>
> The fix for IMPALA-4172/IMPALA-3653 uses Hadoop's Filesystem.listFiles() API to recursively list all files under an HDFS table's parent directory. We then map each file to its corresponding partition. However, the use of listFiles() and the associated code for doing the file-to-partition mapping does not really make sense because listFiles() is just a recursive wrapper around listLocatedStatus(). So for a table with 10k partitions there will be 10k RPCs doing listLocatedStatus().
> We should simplify our code to just loop over all partitions and call listLocatedStatus(). This has the following benefits:
> * Simper code. Would have avoided bugs like IMPALA-4789.
> * Faster code. No need to map files to partitions.
> * Easier to parallelize in the future.
> * Easier to decouple table and partition loading in the future.
> Keep in mind that for S3 tables we do want to use the listFiles() API to avoid being throttled by S3.
> Relevant links:
> https://github.com/apache/hadoop/blob/branch-2.6.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L1720
> https://github.com/apache/hadoop/blob/branch-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L766



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)