You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "xiaojunxiang (Jira)" <ji...@apache.org> on 2024/04/24 07:56:00 UTC
[jira] [Created] (HDFS-17498) Distcp the concat files error, because sourceFS's chesksum is not equals to targetFS's chesksum.
xiaojunxiang created HDFS-17498:
-----------------------------------
Summary: Distcp the concat files error, because sourceFS's chesksum is not equals to targetFS's chesksum.
Key: HDFS-17498
URL: https://issues.apache.org/jira/browse/HDFS-17498
Project: Hadoop HDFS
Issue Type: Improvement
Components: distcp
Affects Versions: 3.3.4
Reporter: xiaojunxiang
Attachments: image-2024-04-24-15-54-16-253.png, image-2024-04-24-15-54-58-047.png, image-2024-04-24-15-55-25-519.png, image-2024-04-24-15-55-48-752.png
When we use distcp, the sourceFS's checksum and targetFS's checksum are checked for consistency after the file transfer is complete.
However, for some files produced by ClientProcotol's concat(RPC method) on the source side, the Block Size is less than 128MB(such as sourceFS file =10MB+10MB, targetFS file = 20MB), so the checksum of the source and destination side will be inconsistent, So It waill cause distcp failed
!image-2024-04-24-15-54-16-253.png!
!image-2024-04-24-15-54-58-047.png!
!image-2024-04-24-15-55-25-519.png!
!image-2024-04-24-15-55-48-752.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org