[GitHub] [hadoop] symious commented on a diff in pull request #5603: HADOOP-18723. Add detail logs if distcp checksum mismatch

["symious (via GitHub)" <gi...@apache.org>]

symious commented on code in PR #5603:
URL: https://github.com/apache/hadoop/pull/5603#discussion_r1185646826


##########
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java:
##########
@@ -596,6 +596,8 @@ public static CopyMapper.ChecksumComparison checksumsAreEqual(
     } else if (sourceChecksum.equals(targetChecksum)) {
       return CopyMapper.ChecksumComparison.TRUE;
     }
+    LOG.info("Checksum not equal. Source checksum: {}, target checksum: {}",
+        sourceChecksum, targetChecksum);

Review Comment:
   In our failed jobs, the mismatch source and target path was printed by https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L633.
   
   Thanks for pointing that out. The paths have also been added to DistCpUtils.



##########
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyCommitter.java:
##########
@@ -569,6 +572,7 @@ private void testCommitWithChecksumMismatch(boolean skipCrc)
                 fs, new Path(sourceBase + srcFilename), null,
                 fs, new Path(targetBase + srcFilename),
                 sourceCurrStatus.getLen()));
+        assertThat(log.getOutput(), containsString("Checksum not equal"));

Review Comment:
   Copied the hamcrest version from "org.apache.hadoop.conf.TestReconfiguration". 
   
   But changed to the Assertions.assertThat one.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org