[jira] [Commented] (HDFS-7553) fix the TestDFSUpgradeWithHA due to BindException

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268218#comment-14268218 ] 

Chris Nauroth commented on HDFS-7553:
-------------------------------------

Hi [~xieliang007].  If I understand correctly, you were trying to reset the port to 0, which would then retrigger the logic for dynamically selecting a new bind port.  However, I don't think this is right, because all of this is happening in a {{Before}} method before each test runs.  The failure report shows the bind failure part-way through one test after a NameNode restart, so it doesn't look like an interaction spanning multiple tests that run into a port conflict with each other.

Another possibility here might be that NameNode shutdown waits for completion of the RPC server shutdown, but not the HTTP server shutdown.  This happens in {{NameNode#join}}:

{code}
  public void join() {
    try {
      rpcServer.join();
    } catch (InterruptedException ie) {
      LOG.info("Caught interrupted exception ", ie);
    }
  }
{code}

However, there is no call to {{HttpServer2#join}}.  This would ultimately block waiting for Jetty's thread pool to shut down.  Perhaps we have a race condition where Jetty remains running bound to that port for a brief window after NameNode shutdown.  Then, when the restart happens, we have a conflict on that port.

> fix the TestDFSUpgradeWithHA due to BindException
> -------------------------------------------------
>
>                 Key: HDFS-7553
>                 URL: https://issues.apache.org/jira/browse/HDFS-7553
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: test
>    Affects Versions: 2.7.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-7553-001.txt
>
>
> see https://builds.apache.org/job/PreCommit-HDFS-Build/9092//testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestDFSUpgradeWithHA/testNfsUpgrade/ :
> Error Message
> Port in use: localhost:57896
> Stacktrace
> java.net.BindException: Port in use: localhost:57896
> 	at sun.nio.ch.Net.bind0(Native Method)
> 	at sun.nio.ch.Net.bind(Net.java:444)
> 	at sun.nio.ch.Net.bind(Net.java:436)
> 	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> 	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> 	at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> 	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:868)
> 	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:704)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:591)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:763)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:747)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1443)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1815)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1796)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testNfsUpgrade(TestDFSUpgradeWithHA.java:285)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)