You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2009/04/09 01:52:12 UTC
[jira] Created: (HADOOP-5644) Namnode is stuck in safe mode
Namnode is stuck in safe mode
-----------------------------
Key: HADOOP-5644
URL: https://issues.apache.org/jira/browse/HADOOP-5644
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.19.1, 0.19.0, 0.18.3, 0.18.2, 0.18.1
Reporter: Suresh Srinivas
Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5644:
-------------------------------------------
Component/s: dfs
Hadoop Flags: [Reviewed]
Good observation!
Your current 0.18 and 0.19 patches still remove the log messages. I guess you have uploaded wrong files.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697564#action_12697564 ]
Hairong Kuang commented on HADOOP-5644:
---------------------------------------
+1. CommitBlockSynchronization() should not persist blocks if sync/append is not supported.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5644:
------------------------------------
Attachment: 5644.branch-0.19.patch
5644.trunk.patch
5644.branch-0.18.patch
# For branches 20 and trunk the block is not persisted in {{FSNamesystem.commitBlockSynchronization()}}, if append support is not enabled. Trunk patch can be applied to branch 20.
# For branches 18 and 19, since there is no flag to enable/disable support for append, block is not persisted without any check for append support.
This needs to be revisited when append functionality is enabled (that is, even with append/sync, if a block is persisted, the datanode must not delete the block from /tmp).
I have tested this fix manually on trunk.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697875#action_12697875 ]
Hairong Kuang commented on HADOOP-5644:
---------------------------------------
+1. The patch looks good.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5644:
------------------------------------
Attachment: 5644.branch-0.19.patch
5644.branch-0.18.patch
Adding {{LOG}} that was deleted in the patch versions for 18 and 19
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698971#action_12698971 ]
Suresh Srinivas commented on HADOOP-5644:
-----------------------------------------
The failed tests are not related to this change.
This functionality is tested manually as it is not easy to automate this test. To test this patch run a command show below:
# Set {{dfs.safemode.threshld.pct}} to 1.000
# Run {{slowout | bin/hadoop fs -put - /test.out}}
where {{slowout}} is a command that outputs text slowly. While the output of the command is being written to ~/test.out, kill the datanode to which client is writing to, followed by the other datanodes to which the file is being pipelined to.
# Restart the cluster. The cluster gets stuck in safe mode without this patch. With this patch, cluster starts up and automatically comes out of safe mode.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697338#action_12697338 ]
Suresh Srinivas commented on HADOOP-5644:
-----------------------------------------
A client is writing data to datanode D1, with pipeline to datanodes D2 and D3. Restarting D1, followed by D2 and D3 results in:
1. File that client was writing gets stuck in the state under construction. Restarting the cluster or lease recovery timeout does not finalize the file.
2. Block that client was writing to is deleted by the datanodes during restart. If the namenode is configured with safemode threshold of 1.0, on restart it does not come out of safemode, since the deleted block is never reported by any datanode.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5644:
------------------------------------
Attachment: 5644.trunk.patch
Attaching the patch for the trunk as the latest attachment.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5644:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Suresh!
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700716#action_12700716 ]
Hudson commented on HADOOP-5644:
--------------------------------
Integrated in Hadoop-trunk #811 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/])
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5644:
------------------------------------
Attachment: 5644.branch-0.19.patch
5644.branch-0.18.patch
Thanks Nicholas. Attaching the right files this time
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697339#action_12697339 ]
Suresh Srinivas commented on HADOOP-5644:
-----------------------------------------
Proposed solution:
When a client encounters a problem, it tries to recover the block. During block recovery, {{FSNamesystem.commitBlockSynchronization()}} the information related to the block is persisted. Namenode during restart, loads the information related to the failed block. The solution is not to persist the block synchronization during block recovery.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698274#action_12698274 ]
Hadoop QA commented on HADOOP-5644:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12405204/5644.trunk.patch
against trunk revision 764287.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/console
This message is automatically generated.
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5644:
------------------------------------
Fix Version/s: 0.21.0
0.20.0
0.19.2
0.18.4
Status: Patch Available (was: Open)
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.1, 0.19.0, 0.18.3, 0.18.2, 0.18.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0
>
> Attachments: 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.18.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.branch-0.19.patch, 5644.trunk.patch
>
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5644) Namnode is stuck in safe mode
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas reassigned HADOOP-5644:
---------------------------------------
Assignee: Suresh Srinivas
> Namnode is stuck in safe mode
> -----------------------------
>
> Key: HADOOP-5644
> URL: https://issues.apache.org/jira/browse/HADOOP-5644
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.