You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/08/23 01:01:08 UTC
[jira] [Comment Edited] (HADOOP-16255) ChecksumFS.Make
FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913832#comment-16913832 ]
Jungtaek Lim edited comment on HADOOP-16255 at 8/23/19 1:00 AM:
----------------------------------------------------------------
[~stevel@apache.org]
Spark also hits this issue https://issues.apache.org/jira/browse/SPARK-28025 though the concern is a bit different. Once "crc for src" is not renamed, it's a leak as we have no opaque way to delete this. We'll just delete the crc file directly as a workaround but definitely looking forward to remove the hack.
Btw, the Hadoop version Spark is using is 2.7.4, but it seems to be a long standing issue through various version lines (maybe all 2.x versions, or even earlier?). What would be the preferred way to do this? Filing new issue describing Spark's case, or could you please add 2.7.4 to the affected version/s here?
was (Author: kabhwan):
[~stevel@apache.org]
Spark also hits this issue https://issues.apache.org/jira/browse/SPARK-28025 though the concern is a bit different. Once "crc for src" is not renamed, it's a leak as we have no opaque way to delete this. We'll just delete the crc file as an workaround but definitely looking forward to remove the hack.
Btw, the Hadoop version Spark is using is 2.7.4, but it seems to be a long standing issue through various version lines (maybe all 2.x versions, or even earlier?). What would be the preferred way to do this? Filing new issue describing Spark's case, or could you please add 2.7.4 to the affected version/s here?
> ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
> ------------------------------------------------------------------------------
>
> Key: HADOOP-16255
> URL: https://issues.apache.org/jira/browse/HADOOP-16255
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 3.1.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum with the file.
> As a result, if a file is renamed over an existing file using rename(src, dest, OVERWRITE) the renamed file will be considered to have an invalid checksum -the old one is picked up instead.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org