You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/03/06 15:08:00 UTC
[jira] [Commented] (HADOOP-15273) distcp to downgrade on checksum
algorithm mismatch to "files unchanged"
[ https://issues.apache.org/jira/browse/HADOOP-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387914#comment-16387914 ]
Steve Loughran commented on HADOOP-15273:
-----------------------------------------
You can't do this, as the current hdfs comparison logic depends on the algorithm name as part of the match; ignoring it would lead to false answers
Better to say "if you don't want checksum comparison, "-skipcrccheck"
> distcp to downgrade on checksum algorithm mismatch to "files unchanged"
> -----------------------------------------------------------------------
>
> Key: HADOOP-15273
> URL: https://issues.apache.org/jira/browse/HADOOP-15273
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Minor
>
> When using distcp without {{-skipCRC}} . If there's a checksum mismatch between src and dest store types (e.g hdfs to s3), then the error message will talk about blocksize, even when its the underlying checksum protocol itself which is the cause for failure
> bq. Source and target differ in block-size. Use -pb to preserve block-sizes during copy. Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By skipping checksums, one runs the risk of masking data-corruption during file-transfer.)
> IF the checksum types are fundamentally different, the error message should say so
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org