You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hrishikesh Gadre (JIRA)" <ji...@apache.org> on 2018/11/06 01:32:00 UTC
[jira] [Comment Edited] (HADOOP-14128) ChecksumFs should override
rename with overwrite flag
[ https://issues.apache.org/jira/browse/HADOOP-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675975#comment-16675975 ]
Hrishikesh Gadre edited comment on HADOOP-14128 at 11/6/18 1:31 AM:
--------------------------------------------------------------------
[~mathieu_c] thanks for reporting the issue and the patch :) I reviewed the patch and the relevant functionality in general and I feel like the code is working as expected (i.e. the root cause of not updating CRC files doesn't seem to be related to rename functionality). To confirm this, I have uploaded a new patch (HADOOP-14128-branch2.8.1-001.patch) which contains ONLY test changes that you proposed in your patch. On my local enviornment, the modified test is passing even without your fix in ChecksumFs.java. The only way to to fail this modified test was to make renameInternal(src, dest, overwrite) method in ChecksumFs.java a NOOP (i.e. not calling method in super class). But since ChecksumFs.java doesn't override renameInternal(src, dest, overwrite) method, I think renameInternal(src, dest, overwrite) method in AbstractFileSystem class is doing the right thing.
Any thoughts? It would be nice to see the test results of my patch as well.
was (Author: hgadre):
[~mathieu_c] thanks for reporting the issue and the patch :) I reviewed the patch and the relevant functionality in general and I feel like the code is working as expected (i.e. the root cause of of not updating CRC files doesn't seem to be related to rename functionality). To confirm this, I have uploaded a new patch (HADOOP-14128-branch2.8.1-001.patch) which contains ONLY test changes that you proposed in your patch. On my local enviornment, the modified test is passing even without your fix in ChecksumFs.java. The only way to to fail this modified test was to make renameInternal(src, dest, overwrite) method in ChecksumFs.java a NOOP (i.e. not calling method in super class). But since ChecksumFs.java doesn't override renameInternal(src, dest, overwrite) method, I think renameInternal(src, dest, overwrite) method in AbstractFileSystem class is doing the right thing.
Any thoughts? It would be nice to see the test results of my patch as well.
> ChecksumFs should override rename with overwrite flag
> -----------------------------------------------------
>
> Key: HADOOP-14128
> URL: https://issues.apache.org/jira/browse/HADOOP-14128
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Affects Versions: 2.8.1
> Reporter: Mathieu Chataigner
> Assignee: Hrishikesh Gadre
> Priority: Major
> Attachments: HADOOP-14128-branch-2.8.1-001.patch, HADOOP-14128-branch2.8.1-001.patch, HADOOP-14128.001.patch, HADOOP-14128.002.patch
>
>
> When I call FileContext.rename(src, dst, Options.Rename.OVERWRITE) on a LocalFs (which extends ChecksumFs), it does not update crc files.
> Every subsequent read on moved files will result in failures due to crc missmatch.
> One solution is to override rename(src, dst, overwrite) the same way it's done with rename(src, dst) and moving crc files accordingly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org