You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "xiaojunxiang (Jira)" <ji...@apache.org> on 2024/04/26 02:54:00 UTC
[jira] [Updated] (HDFS-17498) Distcp the files was generated by concat(RpcMethod) error, because sourceFS's chesksum is not equals to targetFS's chesksum.
[ https://issues.apache.org/jira/browse/HDFS-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xiaojunxiang updated HDFS-17498:
--------------------------------
Summary: Distcp the files was generated by concat(RpcMethod) error, because sourceFS's chesksum is not equals to targetFS's chesksum. (was: Distcp the concat files error, because sourceFS's chesksum is not equals to targetFS's chesksum.)
> Distcp the files was generated by concat(RpcMethod) error, because sourceFS's chesksum is not equals to targetFS's chesksum.
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17498
> URL: https://issues.apache.org/jira/browse/HDFS-17498
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: distcp
> Affects Versions: 3.3.4
> Reporter: xiaojunxiang
> Priority: Major
> Attachments: image-2024-04-24-15-54-16-253.png, image-2024-04-24-15-54-58-047.png, image-2024-04-24-15-55-25-519.png, image-2024-04-24-15-55-48-752.png, image-2024-04-26-10-47-21-884.png, image-2024-04-26-10-47-29-220.png, image-2024-04-26-10-47-49-294.png, image-2024-04-26-10-50-12-238.png, image-2024-04-26-10-50-18-788.png, image-2024-04-26-10-50-23-340.png
>
>
> When we use distcp, the sourceFS's checksum and targetFS's checksum are checked for consistency after the file transfer is complete.
> However, for some files produced by ClientProcotol's concat(RPC method) on the source side, the Block Size is less than 128MB(such as sourceFS file =10MB+10MB, targetFS file = 20MB), so the checksum of the source and destination side will be inconsistent, So It waill cause distcp failed.
>
> Case:
> hadoop fs -put /etc/hosts /tmp/a.txt
> hadoop fs -put /etc/hosts /tmp/b.txt
> hadoop fs -put /etc/hosts /tmp/c.txt
> hadoop fs -concat /tmp/a.txt /tmp/b.txt /tmp/c.txt
> hdfs fsck /tmp/a.txt -files -blocks -locations | grep blk_
> hadoop distcp /tmp/a.txt hdfs://kde-sts-0.com/tmp/res.txt
>
> !image-2024-04-26-10-47-21-884.png!
>
> !image-2024-04-26-10-47-29-220.png!
>
> !image-2024-04-26-10-47-49-294.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org