You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ashangit <gi...@git.apache.org> on 2016/10/26 19:29:28 UTC

[GitHub] spark pull request #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check...

GitHub user ashangit opened a pull request:

    https://github.com/apache/spark/pull/15648

    [SPARK-18119][SPARK-CORE] Namenode safemode check is only performed on one namenode which can stuck the startup of SparkHistory server

    ## What changes were proposed in this pull request?
    
    Instead of using the setSafeMode method that check the first namenode used the one which permitts to check only for active NNs
    
    ## How was this patch tested?
    
    manual tests
    
    Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.
    
    This commit is contributed by Criteo SA under the Apache v2 licence.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/criteo-forks/spark SPARK-18119

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/15648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15648
    
----
commit 501dd4848a2dfd615a44c67274922ca7345f6323
Author: n.fraison <n....@criteo.com>
Date:   2016-10-26T19:11:03Z

    [SPARK-18119][SPARK-CORE] Namenode safemode check is only performed on one namenode which can stuck the startup of SparkHistory server
    
    This commit is contributed by Criteo SA under the Apache v2 licence.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    LGTM, as the javadocs say *If true check only for Active NNs status, else check first NN's status*.  But I don't know enough about HDFS HA to be 
    
    It'll check the first NN, if that is on standby *and stale reads are not allowed( it'll log at error (HDFS-3477 proposes downgrading that), and throw an exception the url [https://s.apache.org/sbnn-error](https://s.apache.org/sbnn-error). If someone sets `dfs.ha.allow.stale.reads"` then they get the old safe mode state; there's nothing that can be done there.
    
    Where my knowledge of HDFS-HA fails is what happens then; Does the RPC client try another NN? Or just it just fail? Maybe @liuml07 could assist there.
    
    The method went in with Hadoop 2.0.3 alpha in [HDFS-3507](https://issues.apache.org/jira/browse/HDFS-3507), so will be across the whole of the Hadoop 2.x line. The enum used did change in 2015, with HDFS-4015; adding `SAFEMODE_FORCE_EXIT`; that action must be avoided. Luckily, the history server isn't trying to exit safemode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    HDFS folks internally indicated this is likely the right change. Let's leave it for a bit before committing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    Merged to master/2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    **[Test build #3437 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3437/consoleFull)** for PR 15648 at commit [`eb76612`](https://github.com/apache/spark/commit/eb7661203a056f4c3c176fcf0dfdc88b6aecfe76).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    CC @steveloughran who may also have good insight into whether this it the right change for HDFS HA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by ashangit <gi...@git.apache.org>.
Github user ashangit commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    Comments taken in account.
    For the test it is in fact a really simple change but let me know if we have to add a it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by ashangit <gi...@git.apache.org>.
Github user ashangit commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    @holdenk and @srowen any news?
    Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by liuml07 <gi...@git.apache.org>.
Github user liuml07 commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    Sorry for coming late. The change is very reasonable. Glad it's merged.
    
    Steve:
    > Where my knowledge of HDFS-HA fails is what happens then; Does the RPC client try another NN? Or just it just fail? Maybe @liuml07 could assist there.
    
    So in general, `setSafeMode(true)` is a READ operation while the `setSafeMode(false)` is an unchecked operation. 
    1. The standby NN will throw an exception for READ operations unless it's configured to allow so via config key `dfs.ha.allow.stale.reads`. And if the StandbyException is caught by the client side, the DFSClient internal proxy will automatically retry multiple times (say, 10) against the other NN (hopefully active). So the `setSafeMode(true)` request will eventually go to the active NN.
    1. In contrast, the `setSafeMode(false)` will go to a NameNode (can be standby) which will return its safe mode status instead of throwing a StandbyException. This return value may fool the client which was expecting the active NN is out of safe mode.
    
    By the way, the above comments make sense only if we're using the logical HDFS service name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    @srowen I'm actually not 100% sure what should happen with hdfs HA -- lemme ask around.  This looks right but worth checking that we're not covering up some other cluster issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15648#discussion_r86467414
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
    @@ -642,7 +642,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     
       // For testing.
    --- End diff --
    
    minor: I think it isn't just for testing anymore, maybe we should remove this comment since we are updating this function anyways?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check is onl...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15648
  
    **[Test build #3437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3437/consoleFull)** for PR 15648 at commit [`eb76612`](https://github.com/apache/spark/commit/eb7661203a056f4c3c176fcf0dfdc88b6aecfe76).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/15648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15648: [SPARK-18119][SPARK-CORE] Namenode safemode check...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15648#discussion_r86467716
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
    @@ -642,7 +642,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     
       // For testing.
       private[history] def isFsInSafeMode(dfs: DistributedFileSystem): Boolean = {
    -    dfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_GET)
    +    dfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_GET, true)
    --- End diff --
    
    Should mention what the true is for (e.g. with a named param or just something like `/* isChecked - run on ActiveNN */`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org