[jira] [Commented] (KUDU-2738) linked_list-test occasionally fails with webserver port bind failure: address already in use

["Adar Dembo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787182#comment-16787182 ] 

Adar Dembo commented on KUDU-2738:
----------------------------------

I don't think it was a separate process because, like you said, we're using a unique loopback IP address and the odds of another application electing to use a non-standard loopback IP address, choosing the same address as ours, and also using the same port are impossibly low.

More likely than not I think we've run afoul of some SO_REUSEADDR behavior w.r.t. TIME_WAIT sockets. According to the test output you attached, the tserver that failed was only started twice. I've listed the parameters involved in choosing the webserver bind socket below.

Start:
{noformat}
--webserver_interface=127.14.25.194
--webserver_port=0
{noformat}

Restart:
{noformat}
--webserver_interface=127.14.25.194
--webserver_port=49008
 {noformat}

I double checked and squeasel does set SO_REUSEADDR on its socket before binding to it (see {{set_ports_option}} in squeasel.c. I don't exactly know how SO_REUSEADDR works, but there's a very informative [SO post|https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t] about it. I suspect that we're being bitten by something here. Maybe it's exacerbated because linked_list-test starts up a PeriodicWebUIChecker which accesses the web UI repeatedly, unlike most other tests. Maybe it's also exacerbated because squeasel sometimes tinkers with the "linger time" of its sockets:
{noformat}
static void close_socket_gracefully(struct sq_connection *conn) {
  struct linger linger;

  // Set linger option to avoid socket hanging out after close. This prevent
  // ephemeral port exhaust problem under high QPS.
  linger.l_onoff = 1;
  linger.l_linger = 1;
  setsockopt(conn->client.sock, SOL_SOCKET, SO_LINGER,
             (char *) &linger, sizeof(linger));

  // Send FIN to the client
  shutdown(conn->client.sock, SHUT_WR);
  set_non_blocking_mode(conn->client.sock);

  // Now we know that our FIN is ACK-ed, safe to close
  closesocket(conn->client.sock);
}
{noformat}

> linked_list-test occasionally fails with webserver port bind failure: address already in use
> --------------------------------------------------------------------------------------------
>
>                 Key: KUDU-2738
>                 URL: https://issues.apache.org/jira/browse/KUDU-2738
>             Project: Kudu
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 1.9.0
>            Reporter: Mike Percy
>            Priority: Trivial
>         Attachments: jenkins_output.txt.gz
>
>
> Occasionally I see linked_list-test fail with the following error on Linux in an automated test environment:
> {code:java}
> E0306 23:35:25.207222 19523 webserver.cc:369] Webserver: set_ports_option: cannot bind to 127.14.25.194:49008: 98 (Address already in use)
> W0306 23:35:25.207244 19523 net_util.cc:457] Trying to use lsof to find any processes listening on 0.0.0.0:49008
> I0306 23:35:25.207249 19523 net_util.cc:460] $ export PATH=$PATH:/usr/sbin ; lsof -n -i 'TCP:49008' -sTCP:LISTEN ; for pid in $(lsof -F p -n -i 'TCP:49008' -sTCP:LISTEN | grep p | cut -f 2 -dp) ; do while [ $pid -gt 1 ] ; do ps h -fp $pid ; stat=($(</proc/$pid/stat)) ; pid=${stat[3]} ; done ; done
> ...
> W0306 23:35:25.583075 19523 net_util.cc:467]
> F0306 23:35:25.583206 19523 tablet_server_main.cc:89] Check failed: _s.ok() Bad status: Runtime error: Webserver: could not start on address 127.14.25.194:49008: set_ports_option: cannot bind to 127.14.25.194:49008: 98 (Address already in use){code}
> I am not sure what would have bound to 0.0.0.0:49008 for a short period of time, or used 126.14.25.194:49008 as an ephemeral address / port pair since it's such a unique loopback IP address.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)