[2/3] incubator-impala git commit: IMPALA-4566: Kudu client glog contention can cause timeouts

[sa...@apache.org]

IMPALA-4566: Kudu client glog contention can cause timeouts

Under stressful workloads, there appears to be significant contention
in glog resulting from Kudu logging, causing timeouts and failed
queries.

An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
WARNINGs grab a lock to flush the log. This is appropriate as Kudu
logs WARNINGs much more frequently than Impala and for things that
Impala would normally consider INFO-level.

Testing: Manually verified that the patch redirects Kudu WARNINGs
to the INFO log. Not tested under stress to verify if this actually
solves the contention problem.

Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Reviewed-on: http://gerrit.cloudera.org:8080/5334
Reviewed-by: Matthew Jacobs <mj...@cloudera.com>
Tested-by: Internal Jenkins


Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/2ae93b03
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/2ae93b03
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/2ae93b03

Branch: refs/heads/master
Commit: 2ae93b03ee00aff84e25db51813c9d1f5cb26b8b
Parents: 02fc53d
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Authored: Fri Dec 2 15:15:55 2016 -0800
Committer: Internal Jenkins <cl...@gerrit.cloudera.org>
Committed: Fri Dec 9 23:28:41 2016 +0000

----------------------------------------------------------------------
 be/src/exec/kudu-util.cc | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/2ae93b03/be/src/exec/kudu-util.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/kudu-util.cc b/be/src/exec/kudu-util.cc
index 5afa0a4..e81baf5 100644
--- a/be/src/exec/kudu-util.cc
+++ b/be/src/exec/kudu-util.cc
@@ -78,14 +78,16 @@ void LogKuduMessage(void* unused, kudu::client::KuduLogSeverity severity,
     const char* filename, int line_number, const struct ::tm* time, const char* message,
     size_t message_len) {
 
-  // Note: we use raw ints instead of the nice LogSeverity typedef
+  // Note: use raw ints instead of the nice LogSeverity typedef
   // that can be found in glog/log_severity.h as it has an import
   // conflict with gutil/logging-inl.h (indirectly imported).
   int glog_severity;
 
   switch (severity) {
     case kudu::client::SEVERITY_INFO: glog_severity = 0; break;
-    case kudu::client::SEVERITY_WARNING: glog_severity = 1; break;
+    // Log Kudu WARNING messages at the INFO level to avoid contention created by glog
+    // locking while flushing WARNING messages.
+    case kudu::client::SEVERITY_WARNING: glog_severity = 0; break;
     case kudu::client::SEVERITY_ERROR: glog_severity = 2; break;
     case kudu::client::SEVERITY_FATAL: glog_severity = 3; break;
     default : DCHECK(false) << "Unexpected severity type: " << severity;