[2/2] impala git commit: IMPALA-6889: Fix race around terminating processes

[lv...@apache.org]

IMPALA-6889: Fix race around terminating processes

A process with empty cmdline can trip up the Process class in our test
code. When a process dies it becomes a zombie until it gets reaped by
its parent.

>From 'man proc':

/proc/[pid]/cmdline
  This read-only file holds the complete command line for the
  process, unless the process is a zombie.  In the latter case,
  there is nothing in this file: that is, a read on this file
  will return 0 characters.  The command-line arguments appear
  in this file as a set of strings separated by null bytes
  ('\0'), with a further null byte after the last string.

To fix this, we take a copy of the cmdline before passing it to the
Process ctor to prevent it from changing.

I couldn't come up with a test for this.

Change-Id: Iecf630e0b71d91469650636e81f940a7bec07113
Reviewed-on: http://gerrit.cloudera.org:8080/10156
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/65bcb160
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/65bcb160
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/65bcb160

Branch: refs/heads/master
Commit: 65bcb1608d0e24603dfd8726a5c6e59e4682717c
Parents: 2fb73f9
Author: Lars Volker <lv...@cloudera.com>
Authored: Mon Apr 23 11:22:39 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Apr 25 21:19:22 2018 +0000

----------------------------------------------------------------------
 tests/common/impala_cluster.py | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/65bcb160/tests/common/impala_cluster.py
----------------------------------------------------------------------
diff --git a/tests/common/impala_cluster.py b/tests/common/impala_cluster.py
index 276c02b..315666b 100644
--- a/tests/common/impala_cluster.py
+++ b/tests/common/impala_cluster.py
@@ -109,12 +109,19 @@ class ImpalaCluster(object):
           if "uid not found" in str(e):
             continue
           raise
-        if process.name == 'impalad' and len(process.cmdline) >= 1:
-          impalads.append(ImpaladProcess(process.cmdline))
-        elif process.name == 'statestored' and len(process.cmdline) >= 1:
-          statestored.append(StateStoreProcess(process.cmdline))
-        elif process.name == 'catalogd' and len(process.cmdline) >= 1:
-          catalogd = CatalogdProcess(process.cmdline)
+        # IMPALA-6889: When a process shuts down and becomes a zombie its cmdline becomes
+        # empty for a brief moment, before it gets reaped by its parent (see man proc). We
+        # copy the cmdline to prevent it from changing between the following checks and
+        # the construction of the *Process objects.
+        cmdline = process.cmdline
+        if len(cmdline) == 0:
+          continue
+        if process.name == 'impalad':
+          impalads.append(ImpaladProcess(cmdline))
+        elif process.name == 'statestored':
+          statestored.append(StateStoreProcess(cmdline))
+        elif process.name == 'catalogd':
+          catalogd = CatalogdProcess(cmdline)
       except psutil.NoSuchProcess, e:
         # A process from get_pid_list() no longer exists, continue.
         LOG.info(e)