[GitHub] [hadoop] kai33 opened a new pull request #919: HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel

[GitBox <gi...@apache.org>]

kai33 opened a new pull request #919: HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel
URL: https://github.com/apache/hadoop/pull/919
 
 
   [HADOOP-16158](https://issues.apache.org/jira/browse/HADOOP-16158)
   
   Copying blocks in parallel (enabled when blocks per chunk > 0) is a great DistCp improvement that can hugely speed up copying big files. 
   But its checksum validation is skipped, e.g. in `RetriableFileCopyCommand.java`
   
   ```
   // isSplit() here is introduced by HADOOP-11794 and 
   // used to indicate if the source data to copy is only a chunk of it 
   // (consists of one or more blocks, not all).
   if (!source.isSplit()) {
     compareCheckSums(sourceFS, source.getPath(), sourceChecksum,
         targetFS, targetPath);
   }
   ```
   
   and this could result in checksum/data mismatch without notifying developers/users (e.g. HADOOP-16049).
   I'd like to provide a patch to add the checksum validation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org