You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Adam Maroti (Jira)" <ji...@apache.org> on 2021/03/29 15:14:00 UTC
[jira] [Updated] (HADOOP-17611) Distcp parallel file copy breaks
the modification time
[ https://issues.apache.org/jira/browse/HADOOP-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Maroti updated HADOOP-17611:
---------------------------------
Description:
The commit HADOOP-11794. Enable distcp to copy blocks in parallel. (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of large files.
In CopyCommitter.java inside concatFileChunks Filesystem.concat is called which changes the modification time therefore the modification times of files copeid by distcp will not match the source files. However this only occurs for large enough files, which are copied by splitting them up by distcp.
In concatFileChunks. before calling concat extract the modification time and apply that to the concatenated resulting file after the concat. (probably best after the rename()).
was:
The commit HADOOP-11794. Enable distcp to copy blocks in parallel. (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of large files.
In CopyCommitter.java inside concatFileChunks Filesystem.concat is called which changes the modification time therefore the modification times of files copeid by distcp will not match the source files. However this only occures for large enough files, which are copied by splitting them up by distcp.
In concatFileChunks. before calling concat extract the modification time and apply that to the concatenated resulting file after the concat. (probably best after the rename()).
> Distcp parallel file copy breaks the modification time
> ------------------------------------------------------
>
> Key: HADOOP-17611
> URL: https://issues.apache.org/jira/browse/HADOOP-17611
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Adam Maroti
> Priority: Major
>
> The commit HADOOP-11794. Enable distcp to copy blocks in parallel. (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of large files.
>
> In CopyCommitter.java inside concatFileChunks Filesystem.concat is called which changes the modification time therefore the modification times of files copeid by distcp will not match the source files. However this only occurs for large enough files, which are copied by splitting them up by distcp.
> In concatFileChunks. before calling concat extract the modification time and apply that to the concatenated resulting file after the concat. (probably best after the rename()).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org