[jira] [Created] (IMPALA-8581) Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable

["Joe McDonnell (JIRA)" <ji...@apache.org>]

Joe McDonnell created IMPALA-8581:
-------------------------------------

             Summary: Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable
                 Key: IMPALA-8581
                 URL: https://issues.apache.org/jira/browse/IMPALA-8581
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 3.3.0
            Reporter: Joe McDonnell


When doing a put or copyFromLocal via the HDFS commandline, it puts the file into a temporary file and then moves it into place. The "-d" option skips the temporary file and copies the file directly into place. On S3, this avoids an extra copy and can also avoid consistency issues. We should look into using the "-d" option across our tests. Our usages of the HDFS commandline generally don't require atomicity of this file copy, so using "-d" is unlikely to impact HDFS tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)