You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hari Mankude (Created) (JIRA)" <ji...@apache.org> on 2012/02/08 02:20:59 UTC
[jira] [Created] (HDFS-2914) HA Standby stuck in safemode when
shared edits directory is bounced
HA Standby stuck in safemode when shared edits directory is bounced
-------------------------------------------------------------------
Key: HDFS-2914
URL: https://issues.apache.org/jira/browse/HDFS-2914
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Hari Mankude
When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209668#comment-13209668 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason.
Hmmmmm, I don't _think_ this should be a problem. We currently support transitioning to the active state while the NN is in safemode, so I don't see why any services would fail to start if we were to enter safemode while transitioning to the active state.
Regardless, even if it is possible, I think you've convinced me that it's not actually necessary.
bq. I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode.
OK, that seems fine. Perhaps we could also have FSNS#startActiveServices interrupt the NameNodeResourceMonitor thread? That would guarantee that a resource check would happen promptly after transitioning to active.
Offline Todd pointed out to me that another thing we could do would be to check for having available resources in the monitorHealth RPC call, which the failover controller can call *before* initiating a failover, to make sure there are available resources on the NN which we want to failover to. That should probably be done in a separate JIRA, though.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291720#comment-13291720 ]
Hudson commented on HDFS-2914:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1071 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1071/])
HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207443#comment-13207443 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
bq. That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior.
Like I mentioned earlier, I would like to open a seperate test jira for this issue.
bq. I don't think that's true. startCommonServices(...) only gets called at NN initialization, not on each failover.
Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203135#comment-13203135 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour?
I think we probably do. If the NFS mount is flaky, we've got bigger problems than just the NN being moved into SM.
bq. The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts.
I guess the admin should configure some alerts for the NN being in SM, then. :)
But regardless, I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes." Such a change would not be specific to the HA branch, however, and should probably be done on trunk.
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Attachment: (was: HDFS-2914-2.0.1.patch)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 2.0.1
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203124#comment-13203124 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
Hi Aaron,
The issue I see is that even if this standby is made active later on, it will not exit out of the safemode unless user does the safemode leave. Do we want this behaviour? The other problem with this approach is that if nfs dir bounces even once, standby will go into safemode and this will happen silently without alerts.
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Attachment: HDFS-2914-trunk.patch
Attaching the Latest patch for trunk(3.0)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 2.0.1
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280794#comment-13280794 ]
Hadoop QA commented on HDFS-2914:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528562/HDFS-2914-2.0.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2505//console
This message is automatically generated.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 2.0.1
>
> Attachments: HDFS-2914-2.0.1.patch, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203169#comment-13203169 ]
Uma Maheswara Rao G commented on HDFS-2914:
-------------------------------------------
{quote}
I could probably be persuaded that the NN should leave SM automatically once resources become available again, as long the implementation includes some measure(s) to prevent the NN from flapping in/out of SM if the free space is hovering near the threshold. Something like "leave SM automatically only if free space is now well above what is required, and only if it's been like that for several minutes."
{quote}
Yes, this sounds good. As NameNodeResourceChecker moved the system into safemode on some condition, should be its responsibility to take out of the safemode whenever system is out of that condition.
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Attachment: (was: HDFS-2914.patch)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-2.0.1.patch, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207441#comment-13207441 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective?
Obviously not a big deal, I'm just always looking for ways to speed up the tests. Feel free to punt on this.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207256#comment-13207256 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. I would rather leave this as is since I could easily make the problem happen with 10s sleep.
I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something?
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA Standby stuck in safemode when
shared edits directory is bounced
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203099#comment-13203099 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
When shared edits dir is bounced, df will return space of zero. Since shared is required dir, standby nn will enter into safe mode.
2012-02-08 01:08:19,850 WARN namenode.NameNodeResourceChecker (NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on volume 'nfs directory' is 0, which is below the configured reserved amount 104857600
2012-02-08 01:08:19,853 WARN namenode.FSNamesystem (FSNamesystem.java:run(3095)) - NameNode low on available disk space. Entering safe mode.
The fix could be trivial enough to exit safe mode when shared resources become available for standby NN.
> HA Standby stuck in safemode when shared edits directory is bounced
> -------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205075#comment-13205075 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
New patch attached with test fixes from comments
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203265#comment-13203265 ]
Jitendra Nath Pandey commented on HDFS-2914:
--------------------------------------------
Standby doesn't need to enter safe mode because it is not writing any transactions anyway. When it transitions to active, that's when a check for available resources to write logs should be performed.
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Target Version/s: 0.24.0 (was: HA branch (HDFS-1623))
Affects Version/s: (was: HA branch (HDFS-1623))
0.24.0
Converting to top-level issue with commit of HDFS-1623.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203897#comment-13203897 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
That sounds fine to me. Since it's not really specific to the shared edits dir per se, let's change the JIRA title to something like "HA: The standby should not enter safemode automatically when resources are low".
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Issue Type: Bug (was: Sub-task)
Parent: (was: HDFS-1623)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Hari Mankude (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude reassigned HDFS-2914:
----------------------------------
Assignee: Hari Mankude
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Hari Mankude (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude updated HDFS-2914:
-------------------------------
Attachment: hdfs-2914
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Fix Version/s: (was: 2.0.1)
3.0.0
Target Version/s: 0.24.0, 2.0.1, 3.0.0 (was: 0.24.0, 2.0.1)
Affects Version/s: 3.0.0
Status: Patch Available (was: Open)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1, 3.0.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 3.0.0
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291642#comment-13291642 ]
Vinay commented on HDFS-2914:
-----------------------------
Thanks Aaron for the review and commit.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291609#comment-13291609 ]
Hudson commented on HDFS-2914:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2409 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2409/])
HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281540#comment-13281540 ]
Hadoop QA commented on HDFS-2914:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528715/HDFS-2914-trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2509//console
This message is automatically generated.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1, 3.0.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 3.0.0
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203962#comment-13203962 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
Hey Hari, a couple quick comments:
# When uploading a patch, please name the file according to the guidelines specified in http://wiki.apache.org/hadoop/HowToContribute. Specifically, this patch file should be named "HDFS-2914-HDFS-1623.patch"
# Please add a test for this new behavior.
# I don't think there's any need to change the resource check interval. Checking available disk space isn't a heavy-weight operation.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Status: Open (was: Patch Available)
Cancelling the recent patc on 2.0.1 branch
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 2.0.1
>
> Attachments: HDFS-2914-2.0.1.patch, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207244#comment-13207244 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira.
That's exactly what I'm talking about, but note that TestHASafeMode doesn't test the case of entering SM because of low resources, which is handled slightly differently in that the NN won't leave SM automatically as it will when SM is entered on startup. Seems like we should add a test for this behavior.
bq. This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s.
I don't think that's true. {{startCommonServices(...)}} only gets called at NN initialization, not on each failover.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Target Version/s: 2.0.1-alpha (was: 0.24.0, 2.0.1-alpha, 3.0.0)
Affects Version/s: (was: 2.0.1-alpha)
(was: 3.0.0)
(was: 0.24.0)
2.0.0-alpha
Fix Version/s: (was: 3.0.0)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205604#comment-13205604 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
Hey Hari, a few more comments:
# The patch file still needs to have the ".patch" extension.
# Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds.
# Our coding conventions require the use of curly braces ("{}") even for single-line if statements.
# What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I *think* the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior.
# Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209201#comment-13209201 ]
Jitendra Nath Pandey commented on HDFS-2914:
--------------------------------------------
bq. Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services.
I was thinking about it a bit, it might get tricky to check for resources when starting active services, because at this point the namenode is still in standby. If it enters safe mode, then if there is any failure in transition we should take care to transition it back to non-safe mode. I am also suspicious that if it transitions to safemode, some active services may not start just because of the safemode, and that would mean loss of service. We cannot throw an exception either, if resources are low, for the same reason.
I am leaning towards separating the two failure (low resources is not a failure though) scenarios, i.e. standby transitions to active irrespective of what its resource status is, and the check for resources is done independently once transition to active is successfully completed. This is consistent with the fact that low resources is not a failure, the cluster is still available in read only mode.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers reassigned HDFS-2914:
------------------------------------
Assignee: Vinay (was: Hari Mankude)
+1, the latest patch looks good to me. I'm going to commit this momentarily.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Summary: HA: Standby stuck in safemode when shared edits directory is bounced (was: HA Standby stuck in safemode when shared edits directory is bounced)
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203955#comment-13203955 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
Uploaded patch. In the patch, I increased the default value of RESOURCE_CHECK_INTERVAL from 5secs to 60secs. 5secs seems to be excessive for this thread to run. If there are issues, I can remove the fix or modify the fix if there is a better number.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208208#comment-13208208 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. Like I mentioned earlier, I would like to open a seperate test jira for this issue.
OK. Please file the JIRA.
bq. Yep, you are right. Since the thread runs every 5 secs, standby->active will go into safemode in 5 secs when resources are low. If shared edits is not available at the time of state transition, active will fail within 5 secs. Let me try this out and update the jira.
Even if the check will necessarily be performed within 5 seconds of becoming active, we might as well run the check during the process of starting active services.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207230#comment-13207230 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
Any further requirements? If not, can this be checked in?
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Status: Resolved (was: Patch Available)
I've just committed this to trunk and branch-2.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276608#comment-13276608 ]
Vinay commented on HDFS-2914:
-----------------------------
Hi all,
According to the current patch proposed, there is no necessity of NameNodeResourceMonitor for standby NN.
that means StandBy will never go to SafeMode because of resource unavailability. In that case, we can start this NNRM as activeService instead of commonService.
Am I missing something..?
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207434#comment-13207434 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
bq. I don't follow this. This seems to only serve to make the test take longer to run, without providing much benefit. Am I missing something?
We are talking about sleeping for 10s vs changing cluster behavior for the nnresourcetracker thread to run every 1s and changing the sleep to 2s and then changing the default of the nnresourctracker thread back to 5s so that it does not distort the remaining portions of the test. Does a 10s sleep vs 2s sleep make a huge difference from a test usability perspective?
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Hari Mankude (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude updated HDFS-2914:
-------------------------------
Summary: HA: Standby should not enter safemode when resources are low (was: HA: Standby stuck in safemode when shared edits directory is bounced)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Attachment: HDFS-2914.patch
Attaching patch for movement of NNRM from CommonService to ActiveService
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291770#comment-13291770 ]
Hudson commented on HDFS-2914:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1104 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1104/])
HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204151#comment-13204151 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
Uploaded patch
@Aaron
1. Sure, changed name.
2. Test added.
3. Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA Standby stuck in safemode when
shared edits directory is bounced
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2914:
---------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HDFS-1623
> HA Standby stuck in safemode when shared edits directory is bounced
> -------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Fix Version/s: 2.0.1
Target Version/s: 0.24.0, 2.0.1 (was: 0.24.0)
Affects Version/s: 2.0.1
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
Submitting patch for 2.0.1 branch
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0, 2.0.1
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Fix For: 2.0.1
>
> Attachments: HDFS-2914-2.0.1.patch, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated HDFS-2914:
------------------------
Attachment: HDFS-2914-2.0.1.patch
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 0.24.0
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-2.0.1.patch, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA Standby stuck in safemode when
shared edits directory is bounced
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203109#comment-13203109 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
Hey Hari, per the discussion on HDFS-1594, it is by design that the NN does not automatically leave SM even after resources become available again. In order to leave SM, the admin can run `hdfs dfsadmin -safemode leave', even while the NN is in the standby state.
> HA Standby stuck in safemode when shared edits directory is bounced
> -------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Hari Mankude (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude updated HDFS-2914:
-------------------------------
Attachment: HDFS-2914-HDFS-1623
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Hari Mankude (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude updated HDFS-2914:
-------------------------------
Attachment: HDFS-2914-HDFS-1623.patch
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Hari Mankude (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude updated HDFS-2914:
-------------------------------
Attachment: HDFS-2914-HDFS-1623
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby stuck in safemode when
shared edits directory is bounced
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203895#comment-13203895 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
I think Jitendra's comment makes sense. Standby will not be writing to edits directory. Making a resource check a nop should be ok for standby.
> HA: Standby stuck in safemode when shared edits directory is bounced
> --------------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205723#comment-13205723 ]
Hari Mankude commented on HDFS-2914:
------------------------------------
bq. The patch file still needs to have the ".patch" extension.
done
bq. Rather than sleep for 10 seconds, let's increase the frequency which the NNResourceChecker threads runs to every 0 or 1 seconds, and then sleep for 2 seconds.
I would rather leave this as is since I could easily make the problem happen with 10s sleep.
bq. Our coding conventions require the use of curly braces ("{}") even for single-line if statements.
done
bq. What do you think the behavior should be for an NN which is active, experiences low resources, then becomes standby? I think the current behavior seems fine (i.e. require the admin to make the now-standby NN leave SM) but I'm wondering if you've considered this case. You might want to write a test case which asserts the desired behavior.
I am not sure that I completely understand your concern. When active has low resources, it goes into safemode. If shared edits goes away, then active dies. If you are talking about doing a switchover (active to standby) when active is in safemode, I thought I saw a test in testHAsafemode for this conditon. If not, I can add a test in a seperate jira.
bq. Note that Jitendra's suggestion also said "When it transitions to active, that's when a check for available resources to write logs should be performed." I agree with this (much as the NN currently checks for available resources on startup) but your patch doesn't implement this.
This is already handled in checkAvailableResources() being called during startupCommonServices(). Also, resourcechecker thread is always running and it will catch the issue in 5s.
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-2914) HA: Standby should not enter safemode
when resources are low
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-2914.
-------------------------------
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-2914, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291607#comment-13291607 ]
Hudson commented on HDFS-2914:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2336 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2336/])
HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291629#comment-13291629 ]
Hudson commented on HDFS-2914:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2356 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2356/])
HDFS-2914. HA: Standby should not enter safemode when resources are low. Contributed by Vinay. (Revision 1347895)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347895
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
> Affects Versions: 2.0.0-alpha
> Reporter: Hari Mankude
> Assignee: Vinay
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623, HDFS-2914-HDFS-1623.patch, HDFS-2914-trunk.patch, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204158#comment-13204158 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
bq. Sure, changed name.
Thanks.
bq. Test added.
Cool. I'll take a look later, unless someone beats me to it.
bq. Well, I think it is still a waste of cpu for a thread to wake up and run a check every 5 secs. However, that can be decided later on and I have removed the configkey update with this patch.
I challenge you to find a benchmark that produces meaningfully improved results with only running the check every 60 seconds instead of every 5. :)
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2914) HA: Standby should not enter
safemode when resources are low
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204912#comment-13204912 ]
Aaron T. Myers commented on HDFS-2914:
--------------------------------------
Hey Hari, a few comments:
# Is the "while (nn1.getNamesystem().isInSafeMode())" actually necessary? I would think this would be covered by the "cluster.waitActive()" call above. If not, then that's a bug in the minicluster that should be fixed.
# It doesn't seem like this test would necessarily reliably fail without the fix, given that the NNResourceChecker thread runs asynchronously and only every few seconds.
# Rather than doing "assertEquals(false, ...)" just do "assertFalse(...)".
> HA: Standby should not enter safemode when resources are low
> ------------------------------------------------------------
>
> Key: HDFS-2914
> URL: https://issues.apache.org/jira/browse/HDFS-2914
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Hari Mankude
> Assignee: Hari Mankude
> Attachments: HDFS-2914-HDFS-1623, hdfs-2914
>
>
> When shared edits dir is bounced, standby NN is put into safemode by the NameNodeResourceMonitor(). However, there is no path for it to exit out of safe mode when shared edits dir reappears.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira