You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Joe McDonnell (JIRA)" <ji...@apache.org> on 2018/04/25 06:12:00 UTC
[jira] [Resolved] (IMPALA-6899) Dataload uses excessive HDFS
commands
[ https://issues.apache.org/jira/browse/IMPALA-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-6899.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.13.0
Impala 3.0
> Dataload uses excessive HDFS commands
> -------------------------------------
>
> Key: IMPALA-6899
> URL: https://issues.apache.org/jira/browse/IMPALA-6899
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 3.0, Impala 2.13.0
>
>
> Dataload has several locations where it does a long string of HDFS commands similar to this:
>
> {code:java}
> hdfs dfs -mkdir bad_table1
> hdfs dfs -put bad_file_1 bad_table1
> hdfs dfs -put bad_file_2 bad_table1
> hdfs dfs -mkdir bad_table2
> hdfs dfs -put bad_file_3 bad_table2
> hdfs dfs -put bad_file_4 bad_table2{code}
> Most hdfs shell commands can take multiple arguments. In particular, "mkdir" can make multiple directories in one command. "put" can copy multiple files into a single destination. This can save on hdfs commandline invocations, which are often expensive due to JVM startup and other costs. For example, the above is equivalent to:
>
> {code:java}
> hdfs dfs -mkdir bad_table1 bad_table2
> hdfs dfs -put bad_file_1 bad_file_2 bad_table1
> hdfs dfs -put bad_file_3 bad_file_4 bad_table2{code}
> Dataload should make these types of optimizations wherever possible.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)