[03/11] impala git commit: IMPALA-6675: Default to --compact_catalog_topic=true.

[ta...@apache.org]

IMPALA-6675: Default to --compact_catalog_topic=true.

Testing:
- Ran a few queries locally
- Ran test_compact_catalog_updates.py locally

Mostafa's perf evaluation:
- 130 node cluster
- Load metadata after invalidate for 4 tables, each
  with 100K partitions and 1 million files

Results compaction on vs. compaction off
- 5.7x reduction in topic size and Network
- 30% reduction in Catalog+Statestore CPU
- 15% speedup in query time
- Compaction of topic takes ~22s in the Catalog
- Time spent by Statestore sending the topics is
  reduced from 90s to 17s
- Max topic update duration reduced from 72s to 11s

Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Reviewed-on: http://gerrit.cloudera.org:8080/9661
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2f508183
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2f508183
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2f508183

Branch: refs/heads/master
Commit: 2f508183cb4dfefe1c67bf8c1988f350700763fe
Parents: b62ecb6
Author: Alex Behm <al...@cloudera.com>
Authored: Thu Mar 15 09:57:10 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Fri Mar 16 19:12:46 2018 +0000

----------------------------------------------------------------------
 be/src/common/global-flags.cc                        |  2 +-
 tests/custom_cluster/test_compact_catalog_updates.py | 10 +++++-----
 2 files changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/2f508183/be/src/common/global-flags.cc
----------------------------------------------------------------------
diff --git a/be/src/common/global-flags.cc b/be/src/common/global-flags.cc
index 3ed0a0f..b62170d 100644
--- a/be/src/common/global-flags.cc
+++ b/be/src/common/global-flags.cc
@@ -88,7 +88,7 @@ DEFINE_bool(abort_on_config_error, true, "Abort Impala startup if there are impr
 DEFINE_bool(disable_mem_pools, false, "Set to true to disable memory pooling. "
     "This can be used to help diagnose memory corruption issues.");
 
-DEFINE_bool(compact_catalog_topic, false, "If true, catalog updates sent via the "
+DEFINE_bool(compact_catalog_topic, true, "If true, catalog updates sent via the "
     "statestore are compacted before transmission. This saves network bandwidth at the"
     " cost of a small quantity of CPU time. Enable this option in cluster with large"
     " catalogs. It must be enabled on both the catalog service, and all Impala demons.");

http://git-wip-us.apache.org/repos/asf/impala/blob/2f508183/tests/custom_cluster/test_compact_catalog_updates.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_compact_catalog_updates.py b/tests/custom_cluster/test_compact_catalog_updates.py
index c824ad5..05252d5 100644
--- a/tests/custom_cluster/test_compact_catalog_updates.py
+++ b/tests/custom_cluster/test_compact_catalog_updates.py
@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-# Test Catalog behavior when --compact_catalog_topic is true.
+# Test Catalog behavior when --compact_catalog_topic is false.
 
 import pytest
 
@@ -23,10 +23,10 @@ from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
 
 class TestCompactCatalogUpdates(CustomClusterTestSuite):
   @pytest.mark.execute_serially
-  @CustomClusterTestSuite.with_args(impalad_args="--compact_catalog_topic=true",
-      catalogd_args="--compact_catalog_topic=true")
-  def test_compact_catalog_topic_updates(self):
-    """Start Impala cluster with compact catalog update topics enabled and run a set
+  @CustomClusterTestSuite.with_args(impalad_args="--compact_catalog_topic=false",
+      catalogd_args="--compact_catalog_topic=false")
+  def test_non_compact_catalog_topic_updates(self):
+    """Start Impala cluster with compact catalog update topics disabled and run a set
     of smoke tests to verify that catalog updates are received properly."""
 
     try: