You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2009/01/08 21:22:59 UTC
[jira] Created: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
-------------------------------------------------------------------
Key: HADOOP-4997
URL: https://issues.apache.org/jira/browse/HADOOP-4997
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.18.0
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Blocker
Fix For: 0.18.3
This is a temporary work around issues discussed in HADOOP-4663.
The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4997:
-------------------------------------
Attachment: deleteTmp-0.18.patch
when the datanode restarts, remove all blocks from the tmp directory of the datanode. However, fsync does not throw an exception.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662205#action_12662205 ]
dhruba borthakur commented on HADOOP-4997:
------------------------------------------
Hi Hairong, I saw ur comment just now about the unit tests. Thanks for fixing it.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-4997:
------------------------------------
Assignee: Raghu Angadi
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi resolved HADOOP-4997.
----------------------------------
Resolution: Fixed
Release Note: Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees.
Hadoop Flags: [Incompatible change]
I just committed this to 0.18 only.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-4997:
------------------------------------
Assignee: (was: Raghu Angadi)
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662177#action_12662177 ]
Hairong Kuang commented on HADOOP-4997:
---------------------------------------
The two junit tests that would fail with this patch are TestFileCreationDelete and TestRenameWhileOpen.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4997:
---------------------------------
Attachment: HADOOP-4997-branch-18.patch
Thanks Hairong.
The updated patch disables the test Hairong mentioned with a very explicit comment that it is a temporary change and it has the original patch Dhruba attached.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664728#action_12664728 ]
Raghu Angadi commented on HADOOP-4997:
--------------------------------------
test-patch :
------------
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
----------
JavaDoc failures are unrelated to this patch (they are for "CyclicIterator.java").
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662163#action_12662163 ]
Hairong Kuang commented on HADOOP-4997:
---------------------------------------
Dhruba, we should also remove the unit tests that test the tmp directory does not get removed after restarting the cluster.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664778#action_12664778 ]
Konstantin Shvachko commented on HADOOP-4997:
---------------------------------------------
Could you please file a bug against CyclicIterator.java
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664642#action_12664642 ]
Konstantin Shvachko commented on HADOOP-4997:
---------------------------------------------
+1 on Raghu's patch. It is critical for 0.18.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4997) workaround for tmp file handling on
DataNodes in 0.18 (HADOOP-4663)
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664785#action_12664785 ]
Raghu Angadi commented on HADOOP-4997:
--------------------------------------
> Could you please file a bug against CyclicIterator.java
done : HADOOP-5077.
> workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
> -------------------------------------------------------------------
>
> Key: HADOOP-4997
> URL: https://issues.apache.org/jira/browse/HADOOP-4997
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch
>
>
> This is a temporary work around issues discussed in HADOOP-4663.
> The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.