[3/3] incubator-impala git commit: IMPALA-3336: qgen: do not randomly generate query options (workaround)

[ta...@apache.org]

IMPALA-3336: qgen: do not randomly generate query options (workaround)

Disabling random query options seems to lessen the effects of
IMPALA-3336. Since much of the memory for IMPALA-3336 is tied up in the
psycopg2 cursor, and these query options are for Impala, there must be
something going on in the query execution path. One theory is that so
many random values for these query options don't make sense, and more
queries are rejected, but meanwhile they continue to run against
Postgres. It's possible the Postgres threads are not cleaning up.

I haven't been able to dive into that, but let's disable this for now.
We're able to get coverage without it. Moreover, I think more care needs
to be given to the values chosen for these query options.

Change-Id: If9c10e98f3be19739e223e89ffd79880e4c31de2
Reviewed-on: http://gerrit.cloudera.org:8080/3712
Tested-by: Michael Brown <mi...@cloudera.com>
Reviewed-by: David Knupp <dk...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/72e3c147
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/72e3c147
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/72e3c147

Branch: refs/heads/master
Commit: 72e3c147e9329b472667c1fa0baeb7c4160e927c
Parents: e2a7038
Author: Michael Brown <mi...@cloudera.com>
Authored: Wed Jul 20 16:39:26 2016 -0700
Committer: Tim Armstrong <ta...@cloudera.com>
Committed: Fri Jul 22 11:03:33 2016 -0700

----------------------------------------------------------------------
 tests/comparison/discrepancy_searcher.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/72e3c147/tests/comparison/discrepancy_searcher.py
----------------------------------------------------------------------
diff --git a/tests/comparison/discrepancy_searcher.py b/tests/comparison/discrepancy_searcher.py
index ed419b9..0ff0fee 100755
--- a/tests/comparison/discrepancy_searcher.py
+++ b/tests/comparison/discrepancy_searcher.py
@@ -230,6 +230,10 @@ class QueryResultComparator(object):
 class QueryExecutor(object):
   '''Concurrently executes queries'''
 
+  # XXX: Set to false while IMPALA-3336 is a problem. Disabling random query options
+  # seems to reduce IMPALA-3336 occurances.
+  ENABLE_RANDOM_QUERY_OPTIONS = False
+
   # If the number of rows * cols is greater than this val, then the comparison will
   # be aborted. Raising this value also raises the risk of python being OOM killed. At
   # 10M python would get OOM killed occasionally even on a physical machine with 32GB
@@ -271,7 +275,7 @@ class QueryExecutor(object):
     # "CREATE VIEW <name> AS ...", this will be the value of "<name>".
     self._table_or_view_name = None
 
-  def set_impala_query_optons(self, cursor):
+  def set_impala_query_options(self, cursor):
     opts = """
         SET MEM_LIMIT={mem_limit};
         SET BATCH_SIZE={batch_size};
@@ -332,8 +336,8 @@ class QueryExecutor(object):
     query_threads = list()
     for sql_writer, cursor, log_file \
         in izip(self.sql_writers, self.cursors, self.query_logs):
-      if cursor.db_type == IMPALA:
-        self.set_impala_query_optons(cursor)
+      if self.ENABLE_RANDOM_QUERY_OPTIONS and cursor.db_type == IMPALA:
+        self.set_impala_query_options(cursor)
       query_thread = Thread(
           target=self._fetch_sql_results,
           args=[query, cursor, sql_writer, log_file],