You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2017/02/25 02:53:44 UTC
[jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask
when src is a directory
[ https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883969#comment-15883969 ]
Hive QA commented on HIVE-14864:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854547/HIVE-14864.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10259 tests executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3768/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3768/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3768/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854547 - PreCommit-HIVE-Build
> Distcp is not called from MoveTask when src is a directory
> ----------------------------------------------------------
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Sahil Takiar
> Attachments: HIVE-14864.1.patch, HIVE-14864.2.patch, HIVE-14864.3.patch, HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() > conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
> LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes. (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) + ")");
> LOG.info("Launch distributed copy (distcp) job.");
> HiveConfUtil.updateJobCredentialProviders(conf);
> copied = shims.runDistCp(src, dst, conf);
> if (copied && deleteSource) {
> srcFS.delete(src, true);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)