You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2017/09/11 05:40:01 UTC
[jira] [Commented] (HADOOP-12326) Implement
ChecksumFileSystem#getFileChecksum equivalent to HDFS for easy check
[ https://issues.apache.org/jira/browse/HADOOP-12326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160759#comment-16160759 ]
Hadoop QA commented on HADOOP-12326:
------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HADOOP-12326 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-12326 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12752394/HADOOP-12326.007.patch |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13251/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implement ChecksumFileSystem#getFileChecksum equivalent to HDFS for easy check
> ------------------------------------------------------------------------------
>
> Key: HADOOP-12326
> URL: https://issues.apache.org/jira/browse/HADOOP-12326
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 2.7.1
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: HADOOP-12326.001.patch, HADOOP-12326.002.patch, HADOOP-12326.003.patch, HADOOP-12326.004.patch, HADOOP-12326.005.patch, HADOOP-12326.007.patch
>
>
> If we have same-content files, one local and one remotely on HDFS (after downloading or uploading), getFileChecksum can provide a quick check whether they are consistent. To this end, we can switch to CRC32C on local filesystem. The difference in block sizes does not matter, because for the local filesystem it's just a logical parameter.
> {code}
> $ hadoop fs -Dfs.local.block.size=134217728 -checksum file:${PWD}/part-m-00000 part-m-00000
> 15/08/15 13:30:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> file:///Users/gshegalov/workspace/hadoop-common/part-m-00000 MD5-of-262144MD5-of-512CRC32C 000002000000000000040000e84fb07f8c9d4ef3acb5d1983a7e2a68
> part-m-00000 MD5-of-262144MD5-of-512CRC32C 000002000000000000040000e84fb07f8c9d4ef3acb5d1983a7e2a68
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org