[jira] [Commented] (HUDI-1401) Presto use of Metadata Table for file listings

["Udit Mehrotra (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244297#comment-17244297 ] 

Udit Mehrotra commented on HUDI-1401:
-------------------------------------

At a high level, to support metadata based file listing with Presto we will have to inject another implementation for listing the files within each partition [here|https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/HadoopDirectoryLister.java#L45]. Here presto needs *LocatedFileStatus* instead of the regular *FileStatus*. LocatedFileStatus also stored *blockLocations* and it seems like Presto needs it, and if is not available it leads to extra RPC calls to name node which is what we tried to solve with the [PathFilter approach|https://prestodb.io/blog/2020/08/04/prestodb-and-hudi#moving-away-from-inputformatgetsplits]. Given this, I believe to have this optimization we will have to store block locations in the metadata table as well, and keep it updated as well. While I don't see this causing any issues for S3 but for HDFS it might be something to consider. Would like to get some thoughts on if this is a blocker for HDFS ? 

Also, based on recent discussion we are also planning to get rid of the PathFilter for presto and instead directly use the FileSystemView within Presto code for filtering of latest commit files. The reason is that PathFilter is applied to each and every file, and Presto does all this only at the presto co-ordinator and it is a major bottleneck based on a recent investigation we did for an EMR customer. Ultimately we disabled the PathFilter and were able to obtain better performance through InputFormat getSplits approach.

cc [~vinoth] [~bhasudha] [~pwason]

> Presto use of Metadata Table for file listings
> ----------------------------------------------
>
>                 Key: HUDI-1401
>                 URL: https://issues.apache.org/jira/browse/HUDI-1401
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: Presto Integration
>            Reporter: Vinoth Chandar
>            Assignee: Udit Mehrotra
>            Priority: Blocker
>             Fix For: 0.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)