[jira] [Comment Edited] (HIVE-15546) Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel

["t oo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804873#comment-16804873 ] 

t oo edited comment on HIVE-15546 at 4/1/19 8:01 AM:
-----------------------------------------------------

[~stakiar] Issue still faced with single threading - HIVE-21546


was (Author: toopt4):
Did this ever make release 2.3? I can't see it in [https://github.com/apache/hive/blob/rel/release-2.3.0/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java] and issue still faced with single threading (https://stackoverflow.com/questions/55416703/hiveserver2-on-spark-mapred-fileinputformat-total-input-files-to-process)

> Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-15546
>                 URL: https://issues.apache.org/jira/browse/HIVE-15546
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>             Fix For: 2.3.0
>
>         Attachments: HIVE-15546.1.patch, HIVE-15546.2.patch, HIVE-15546.3.patch, HIVE-15546.4.patch, HIVE-15546.5.patch, HIVE-15546.6.patch
>
>
> When running on blobstores (like S3) where metadata operations (like listStatus) are costly, Utilities.getInputPaths() can add significant overhead when setting up the input paths for an MR / Spark / Tez job.
> The method performs a listStatus on all input paths in order to check if the path is empty. If the path is empty, a dummy file is created for the given partition. This is all done sequentially. This can be really slow when there are a lot of empty partitions. Even when all partitions have input data, this can take a long time.
> We should either:
> (1) Just remove the logic to check if each input path is empty, and handle any edge cases accordingly.
> (2) Multi-thread the listStatus calls



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)