You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by "Korb, Michael [USA]" <Ko...@bah.com> on 2013/12/18 06:43:24 UTC

HDFS Data Integrity in copyToLocal

Hello,

How can I verify the integrity of files copied to local from HDFS? Does HDFS store MD5s of full files anywhere? From what I can find, FileSystem.getFileChecksum() is relevant to replication and not comparison across filesystems (http://stackoverflow.com/questions/14563245/hdfs-file-checksum).

The Data Integrity section in HDFS Architecture (http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) does not make it clear if, or how, copyToLocal verifies the integrity of the copied file.

Thanks,
Mike