You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2007/03/07 05:09:24 UTC
[jira] Created: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Periodic checkpointing cannot resume if the secondary name-node fails.
----------------------------------------------------------------------
Key: HADOOP-1076
URL: https://issues.apache.org/jira/browse/HADOOP-1076
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: Konstantin Shvachko
If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
"edits" - is the one which current checkpoint is to be based upon.
"edits.new" - is where new name space edits are currently logged.
The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
In fact the primary node will be throwing an exception complaining about the existing "edits.new"
There is only one way to get rid of the edits.new file - to restart the primary name-node.
So in a way if secondary name-node fails then you should restart the whole cluster.
Here is a rather simple modification to the current approach, which we discussed with Dhruba.
When secondary node requests to rollEditLog() the primary node should roll the edit log only if
it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
and the primary node will keep accumulating new edits in the "edits.new".
In order to make it work the primary node should also ignore any rollFSImage() requests when it
already started to perform one. Otherwise the new image can become corrupted if two secondary
nodes request to rollFSImage() at the same time.
2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
that are not used anywhere. We should remove them and others alike if found.
Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Status: Patch Available (was: Open)
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: secondaryRestart3.patch
merged patch with latest trunk.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart3.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: (was: secondaryRestart.patch)
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart2.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1076) Periodic checkpointing cannot
resume if the secondary name-node fails.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530243 ]
Hudson commented on HADOOP-1076:
--------------------------------
Integrated in Hadoop-Nightly #250 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/250/])
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: (was: secondaryRestart3.patch)
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1076) Periodic checkpointing cannot
resume if the secondary name-node fails.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529964 ]
Hadoop QA commented on HADOOP-1076:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366319/secondaryRestart4.patch
against trunk revision r578879.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/815/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/815/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/815/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/815/console
This message is automatically generated.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Fix Version/s: 0.15.0
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.15.0
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: secondaryRestart4.patch
Incorporated all of Konstantin's comments.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: secondaryRestart.patch
This patch allows the secondary namenode to restart without restarting the primary namenode.
If rollEditLog finds that the edits log already exists, then it simply returns success. rollFsImage fails if it was not preceeded by a call to rollEditLog. This lock-step ensures that a stale instance of a secondary namenode cannot fool the primary namenode into uploading a stale fsimage file.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.15.0
>
> Attachments: secondaryRestart.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-1076:
----------------------------------------
Assignee: dhruba borthakur
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: (was: secondaryRestart2.patch)
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart3.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1076) Periodic checkpointing cannot
resume if the secondary name-node fails.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529593 ]
Konstantin Shvachko commented on HADOOP-1076:
---------------------------------------------
+1
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Attachment: secondaryRestart2.patch
The periodic checkpoint protocol is changed to handle the case if two Secondary's are racing with one another to upload a new checkpoint.
The NameNode periodic checkpoint has four states. A rollEdit moves the state to ROLLED_EDIT. A upload new image is allowed only if the state is ROLLED_EDIT. It sets the state to UPLOAD_START. When the upload of the upload of the new image is finished, the state is set to UPLOAD_DONE. The rollFsImage is allowed only if the state is UPLOAD_DONE.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart2.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1076) Periodic checkpointing cannot resume
if the secondary name-node fails.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1076:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart4.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1076) Periodic checkpointing cannot
resume if the secondary name-node fails.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528959 ]
Konstantin Shvachko commented on HADOOP-1076:
---------------------------------------------
# Need to increment ClientProtocol.versionID
{code}
* 19 : rollEditLog() returns a token to uniquely identify the editfile.
*/
public static final long versionID = 18L;
{code}
# Constants should be all capital: FSNamesystem.DATE_FORM instead of
{code}
private static final SimpleDateFormat dateform =
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
{code}
# FSEditLog.loadFSEdits(File edits) : 396 -- local variable is never read.
{code}
DatanodeDescriptor node = fsNamesys.getDatanode(nodeID);
{code}
# GetImageServlet: redundant imports
{code}
import org.apache.hadoop.conf.*;
import org.apache.hadoop.mapred.StatusHttpServer;
{code}
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart3.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1076) Periodic checkpointing cannot
resume if the secondary name-node fails.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524955 ]
Konstantin Shvachko commented on HADOOP-1076:
---------------------------------------------
I managed to corrupt current name-node image using your patch.
Actually the image was set to an empty file, so that the name-node would not even restart after the checkpoint.
I started 2 secondary nodes. The first of them was in the middle of getFSImage(), when the second called rollFSImage()
and received the following exception:
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212)
at org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375)
at org.apache.hadoop.dfs.SecondaryNameNode$GetImageServlet.doGet(SecondaryNameNode.java:455)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
This is likely to be related to the patch, since the second secondary node would just get an exception trying to rollEditsLog().
My guess is that your patch prohibits to rollFSImage() if the edits log was not rolled, instead of
prohibiting 2 simultaneous rollFSImage().
> Periodic checkpointing cannot resume if the secondary name-node fails.
> ----------------------------------------------------------------------
>
> Key: HADOOP-1076
> URL: https://issues.apache.org/jira/browse/HADOOP-1076
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: dhruba borthakur
> Fix For: 0.15.0
>
> Attachments: secondaryRestart.patch
>
>
> If secondary name-node fails during checkpointing then the primary node will have 2 edits file.
> "edits" - is the one which current checkpoint is to be based upon.
> "edits.new" - is where new name space edits are currently logged.
> The problem is that the primary node cannot do checkpointing until "edits.new" file is in place.
> That is, even if the secondary name-node is restarted periodic checkpointing is not going to be resumed.
> In fact the primary node will be throwing an exception complaining about the existing "edits.new"
> There is only one way to get rid of the edits.new file - to restart the primary name-node.
> So in a way if secondary name-node fails then you should restart the whole cluster.
> Here is a rather simple modification to the current approach, which we discussed with Dhruba.
> When secondary node requests to rollEditLog() the primary node should roll the edit log only if
> it has not been already rolled. Otherwise the existing "edits" file will be used for checkpointing
> and the primary node will keep accumulating new edits in the "edits.new".
> In order to make it work the primary node should also ignore any rollFSImage() requests when it
> already started to perform one. Otherwise the new image can become corrupted if two secondary
> nodes request to rollFSImage() at the same time.
> 2. Also, after the periodic checkpointing patch HADOOP-227 I see pieces of unusable code.
> I noticed one data member SecondaryNameNode.localName and at least 4 methods in FSEditLog
> that are not used anywhere. We should remove them and others alike if found.
> Supporting unusable code is such a waist of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.