You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2009/09/25 02:56:15 UTC
[jira] Created: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
HDFS needs to support new rename introduced for FileContext
-----------------------------------------------------------
Key: HDFS-654
URL: https://issues.apache.org/jira/browse/HDFS-654
Project: Hadoop HDFS
Issue Type: New Feature
Affects Versions: 0.21.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Patch Available (was: Open)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Open (was: Patch Available)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769369#action_12769369 ]
Hadoop QA commented on HDFS-654:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423032/hdfs-654.9.patch
against trunk revision 828926.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 25 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/56/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/56/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/56/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/56/console
This message is automatically generated.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760115#action_12760115 ]
dhruba borthakur commented on HDFS-654:
---------------------------------------
Would appreciate it a lot if you can please explain in short what the new semantics would be. what happens if the destination directory exists? what if the destination directory is non-empty? what if the destination is already a file.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761789#action_12761789 ]
Konstantin Shvachko commented on HDFS-654:
------------------------------------------
# Parameter type {{Options.Rename}} would look better in {{rename()}} methods.
# Increment {{ClientProtocol}} version.
# import warning in FSDirectory.
# Decrement {{LAYOUT_VERSION}} as you introduce new journal record.
# I think that the whole stack of old {{rename()}} methods should be deprecated starting from {{DistributedFileSystem.rename(src, dst)}} down to {{FSDirectory.unprotectedRenameTo()}}. Because otherwise people writing tests or pure HDFS applications will not know which rename to use, and may by mistake use the one with the old semantics, which is not desirable. It will also clarify why you did not reuse code for 2 renames, but rather built a whole new branch of methods for the new semantics.
# It is better to put comments related to deprecation in JavaDoc under {{/** @deprecated **/}} section.
# {{FSEditLog.OP_RENAME}} should be used for the new semantics. The old rename operation should be called {{FSEditLog.OP_RENAME_OLD}} so that we could remove it later.
# You should probably verify that old layout images do not contain new rename op.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Open (was: Patch Available)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: HDFS-654.patch
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.7.patch
New patch has the following additional changes:
# Made changes to fix the rename issue in HDFS-677. Before removing src and dst during rename quota is verified. On any failure adding src and dst back is done without checking for quota.
# RenameAspects.aj which interecepted the calls to FSDirectory.addChild() now intercepts FSDirectory.addChildNoQuotaCheck(). Instead of checked QuotaExceedsException, which cannot be thrown since addChildNoQuotaCheck() does not throw exception, RuntimeException is thrown.
# Addressed the comment from Nicholas.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.5.patch
Latest patch passes all the tests. Here is the test-patch result:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 16 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Open (was: Patch Available)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.3.patch
New patch incorporates all the comments from Konstantin, except deprecation. After hudson tests to confirm the patch is good, I will attach another patch with old Rename deprecated. This will result in warnings. These need to be ignored to submit the patch.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762879#action_12762879 ]
Hadoop QA commented on HDFS-654:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421387/hdfs-654.5.patch
against trunk revision 822153.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 16 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/56/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/56/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/56/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/56/console
This message is automatically generated.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760130#action_12760130 ]
Suresh Srinivas commented on HDFS-654:
--------------------------------------
Dhruba this is as per the discussion in HADOOP-6240. This introduced a non atomic new Rename, as described in FileSystem#rename(String, String, Rename...). This change is overrides the default implementation. In addition to new semantics, rename will be atomic.
bq. Maybe I missed something, but does this do the right accounting for calculating the total number of inodes in the filesystem(to ensure correct working of namespace quotas)?
can you be more elaborate? I did not understand the concern.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760119#action_12760119 ]
dhruba borthakur commented on HDFS-654:
---------------------------------------
Maybe I missed something, but does this do the right accounting for calculating the total number of inodes in the filesystem(to ensure correct working of namespace quotas)?
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763267#action_12763267 ]
Tsz Wo (Nicholas), SZE commented on HDFS-654:
---------------------------------------------
Talked to Jakob: We have to update TestOfflineImageViewer when FSConstants.LAYOUT_VERSION is changed.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Open (was: Patch Available)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769294#action_12769294 ]
Suresh Srinivas commented on HDFS-654:
--------------------------------------
I have made all the changes suggested. Here are my comments:
> 1. FSNamesystem.renameTo() should not be public.
Currently all methods called from NameNode are public. I followed the same pattern. I have change renameTo methods both old and new to package private.
> 5. Is OIV will be able to read new rename edit log records? Just adding the new supported version probably does not fix the reading problem by itself.
Per Jakob, the OIV reads FSImage only and not edit log. It needs to know all the supported versions. Without updating it, it will not load the FSImage with the new version.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769375#action_12769375 ]
Konstantin Shvachko commented on HDFS-654:
------------------------------------------
That is right OIV does not care about edits log.
+1 patch looks great.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Patch Available (was: Open)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Patch Available (was: Open)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.9.patch
New patch.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769017#action_12769017 ]
Konstantin Shvachko commented on HDFS-654:
------------------------------------------
# {{FSNamesystem.renameTo()}} should not be public.
# format new methods to fit 80 symbol boundary.
# In the new {{FSDirectory.unprotectedRenameTo()}} if {{collectSubtreeBlocksAndClear()}} throws a RuntimeException in line 649, then frinalized section will try to put {{removedDst}} back in the treeeven though there the {{renamedSrc}} is already there. I propose to move the assignment "{{removedDst = null;}}" three lines up.
# In {{FSEditLog.loadEditRecords()}} you should follow the same naming convention with statistical variables {{numOpRename}} as with the operation constants:
{{numOpRename}} should count new renames, and {{numOpRenameOld}} - the old ones.
This is only a log message, but it is going to be confusing otherwise.
# Is OIV will be able to read new rename edit log records? Just adding the new supported version probably does not fix the reading problem by itself.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.5.patch
New patch with deprecated methods as suggested by Konstantin. In three places deprecated warnings are suppressed. First one in FSEditLog which calls deprecated rename to replay a change recorded in the edits. Two others are in tests.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769593#action_12769593 ]
Hudson commented on HDFS-654:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #59 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/59/])
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760132#action_12760132 ]
dhruba borthakur commented on HDFS-654:
---------------------------------------
Thanks for the explanation. I was referring to the check in FSNameSystem.checkFsObjectLimit() that checks the total number of inodes in the FS. I was worried that you allow replacing a non-empty directory with an empty directory in which case the totalInode in the FS has to be adjusted accordingly.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Fix Version/s: 0.22.0
Status: Patch Available (was: Open)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768388#action_12768388 ]
Suresh Srinivas commented on HDFS-654:
--------------------------------------
Failed test TestFileAppend2.testComplexAppend is unrelated to this change.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Status: Patch Available (was: Open)
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768124#action_12768124 ]
Hadoop QA commented on HDFS-654:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422588/hdfs-654.7.patch
against trunk revision 826905.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 25 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/46/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/46/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/46/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/46/console
This message is automatically generated.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.2.patch
Attaching new patch for the latest trunk that includes append changes
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770274#action_12770274 ]
Hudson commented on HDFS-654:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #78 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/78/])
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762360#action_12762360 ]
Suresh Srinivas commented on HDFS-654:
--------------------------------------
> FSNameSystem.checkFsObjectLimit()
The count changes when the destination is removed. {{FSDirectory.removeChild(dstInode)}} and {{FSNamesystem.removePathAndBlocks()}} decrements the total INode count and the number of blocks. Also the lease to the removed destination is also removed.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Attachment: hdfs-654.1.patch
New patch.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-654:
---------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed the patch to trunk.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767676#action_12767676 ]
Hadoop QA commented on HDFS-654:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422588/hdfs-654.7.patch
against trunk revision 826905.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 25 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/40/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/40/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/40/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/40/console
This message is automatically generated.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761791#action_12761791 ]
Konstantin Shvachko commented on HDFS-654:
------------------------------------------
> FSNameSystem.checkFsObjectLimit()
I don't see that we check the total object limit in the old rename() and we don't do it in the new one. Should we?
To me rename does not change the total number of objects in the system if it is atomic. So probably we should not.
The directory quotes should be updated though. I think it is taken care of. Suresh could you please confirm this.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769823#action_12769823 ]
Hudson commented on HDFS-654:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #120 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/120/])
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-654) HDFS needs to support new rename
introduced for FileContext
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769568#action_12769568 ]
Hudson commented on HDFS-654:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #82 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/82/])
. Add support new atomic rename functionality in HDFS for supporting rename in FileContext. Contributed by Suresh Srinivas.
> HDFS needs to support new rename introduced for FileContext
> -----------------------------------------------------------
>
> Key: HDFS-654
> URL: https://issues.apache.org/jira/browse/HDFS-654
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-654.1.patch, hdfs-654.2.patch, hdfs-654.3.patch, hdfs-654.5.patch, hdfs-654.5.patch, hdfs-654.7.patch, hdfs-654.9.patch, HDFS-654.patch
>
>
> New rename functionality with different semantics to overwrite the existing destination was introduced for use in FileContext. Currently the default implementation in FileSystem is not atomic. This change implements atomic rename operation for use by FileContext.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.