You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Dennis Huo (JIRA)" <ji...@apache.org> on 2018/02/01 01:18:00 UTC
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in
HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347851#comment-16347851 ]
Dennis Huo commented on HDFS-13056:
-----------------------------------
Uploaded initial end-to-end working draft against trunk which supports CRC32/CRC32C and partial file prefixes of arbitrary bytes-per-crc or blocksize and across replicated vs striped encodings as well.
Still a TODO to support the striped-reconstruction path, and adding stripe support made everything a lot messier so some refactoring is in order. Also, unittests still pending, but manual testing in a real setup works:
{code:java}
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-default1.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-default2.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-bpc1024.dat
$ hdfs dfs -Ddfs.blocksize=67108864 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-blocksize64mb.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat hdfs:///tmp/random-crctest-unaligned1.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat hdfs:///tmp/random-crctest-unaligned2.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-gzipcrc32-1.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest-gzipcrc32-2.dat
$ hdfs dfs -mkdir hdfs:///tmpec
$ hdfs ec -enablePolicy -policy XOR-2-1-1024k
$ hdfs ec -setPolicy -path hdfs:///tmpec -policy XOR-2-1-1024k
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-default1.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-default2.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-bpc1024.dat
$ hdfs dfs -Ddfs.blocksize=67108864 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-blocksize64mb.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat hdfs:///tmpec/random-crctest-unaligned1.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat hdfs:///tmpec/random-crctest-unaligned2.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-gzipcrc32-1.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -Ddfs.bytes-per-checksum=1024 -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmpec/random-crctest-gzipcrc32-2.dat
$ hdfs dfs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest-blocksize64mb.dat MD5-of-131072MD5-of-512CRC32C 0000020000000000000200008baa940ef6ed21fb4bd6224ce917d127
hdfs:///tmp/random-crctest-bpc1024.dat MD5-of-131072MD5-of-1024CRC32C 000004000000000000020000930b0d7ad333786a839b044ed8d18d2d
hdfs:///tmp/random-crctest-default1.dat MD5-of-262144MD5-of-512CRC32C 000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest-default2.dat MD5-of-262144MD5-of-512CRC32C 000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest-gzipcrc32-1.dat MD5-of-262144MD5-of-512CRC32 00000200000000000004000049d52fdd25aa08559e20536acc34d51d
hdfs:///tmp/random-crctest-gzipcrc32-2.dat MD5-of-131072MD5-of-1024CRC32 0000040000000000000200001d5468ea4093ddb3741790b8dc3b9a57
hdfs:///tmp/random-crctest-unaligned1.dat MD5-of-262144MD5-of-512CRC32C 0000020000000000000400000da665dadca0df00456206f234d5f8b0
hdfs:///tmp/random-crctest-unaligned2.dat MD5-of-131072MD5-of-1024CRC32C 00000400000000000002000027c2198f48224a0ddb92c4dc4addd28b
$ hdfs dfs -checksum hdfs:///tmpec/random-crctest*.dat
18/02/01 01:15:54 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.2-hadoop2
hdfs:///tmpec/random-crctest-blocksize64mb.dat MD5-of-131072MD5-of-512CRC32C 0000020000000000000200005b54faaa368ed81b25984a746c767d39
hdfs:///tmpec/random-crctest-bpc1024.dat MD5-of-131072MD5-of-1024CRC32C 00000400000000000002000089a128b1e1995256bdb34fb95720dafc
hdfs:///tmpec/random-crctest-default1.dat MD5-of-262144MD5-of-512CRC32C 00000200000000000004000007ee18e8f4909647adf085ec0f464d1a
hdfs:///tmpec/random-crctest-default2.dat MD5-of-262144MD5-of-512CRC32C 00000200000000000004000007ee18e8f4909647adf085ec0f464d1a
hdfs:///tmpec/random-crctest-gzipcrc32-1.dat MD5-of-262144MD5-of-512CRC32 000002000000000000040000d79ad1fa00fad2f0adb18f49f2e90bb3
hdfs:///tmpec/random-crctest-gzipcrc32-2.dat MD5-of-131072MD5-of-1024CRC32 000004000000000000020000126ac7bc467c59942734bd8ebf690440
hdfs:///tmpec/random-crctest-unaligned1.dat MD5-of-262144MD5-of-512CRC32C 0000020000000000000400004b95df26144cba3d1a0ab87cea048b66
hdfs:///tmpec/random-crctest-unaligned2.dat MD5-of-131072MD5-of-1024CRC32C 000004000000000000020000c8b50f1216f55608975624f6a34542bc
$ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum hdfs:///tmp/random-crctest*.dat
18/02/01 01:15:57 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.2-hadoop2
hdfs:///tmp/random-crctest-blocksize64mb.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-bpc1024.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-default1.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-default2.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-gzipcrc32-1.dat COMPOSITE-CRC32 721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-gzipcrc32-2.dat COMPOSITE-CRC32 721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-unaligned1.dat COMPOSITE-CRC32C c3842f6100000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-unaligned2.dat COMPOSITE-CRC32C c3842f6100000000000000000000000000000000000000000000000000000000
$ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum hdfs:///tmpec/random-crctest*.dat
18/02/01 01:16:00 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.2-hadoop2
hdfs:///tmpec/random-crctest-blocksize64mb.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-bpc1024.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-default1.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-default2.dat COMPOSITE-CRC32C 4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-gzipcrc32-1.dat COMPOSITE-CRC32 721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-gzipcrc32-2.dat COMPOSITE-CRC32 721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-unaligned1.dat COMPOSITE-CRC32C c3842f6100000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-unaligned2.dat COMPOSITE-CRC32C c3842f6100000000000000000000000000000000000000000000000000000000
{code}
> Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-13056
> URL: https://issues.apache.org/jira/browse/HDFS-13056
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, erasure-coding, federation, hdfs
> Affects Versions: 3.0.0
> Reporter: Dennis Huo
> Priority: Major
> Attachments: HDFS-13056-branch-2.8.001.patch, HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf, hdfs-file-composite-crc32-v2.pdf
>
>
> FileChecksum was first introduced in [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are already stored as part of datanode metadata, and the MD5 approach is used to compute an aggregate value in a distributed manner, with individual datanodes computing the MD5-of-CRCs per-block in parallel, and the HDFS client computing the second-level MD5.
>
> A shortcoming of this approach which is often brought up is the fact that this FileChecksum is sensitive to the internal block-size and chunk-size configuration, and thus different HDFS files with different block/chunk settings cannot be compared. More commonly, one might have different HDFS clusters which use different block sizes, in which case any data migration won't be able to use the FileChecksum for distcp's rsync functionality or for verifying end-to-end data integrity (on top of low-level data integrity checks applied at data transfer time).
>
> This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 during the addition of checksum support for striped erasure-coded files; while there was some discussion of using CRC composability, it still ultimately settled on hierarchical MD5 approach, which also adds the problem that checksums of basic replicated files are not comparable to striped files.
>
> This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses CRC composition to remain completely chunk/block agnostic, and allows comparison between striped vs replicated files, between different HDFS instances, and possible even between HDFS and other external storage systems. This feature can also be added in-place to be compatible with existing block metadata, and doesn't need to change the normal path of chunk verification, so is minimally invasive. This also means even large preexisting HDFS deployments could adopt this feature to retroactively sync data. A detailed design document can be found here: https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org