[impala] branch master updated (2e7f689 -> 719ac4a)

[ta...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.


    from 2e7f689  IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
     new 8f533e3  Allow data cache to be enabled optionally when running tests
     new 719ac4a  IMPALA-8121: part 3: invalidate on memory pressure

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 bin/run-all-tests.sh       | 15 +++++++++++++++
 docker/catalogd/Dockerfile |  3 ++-
 2 files changed, 17 insertions(+), 1 deletion(-)

[impala] 02/02: IMPALA-8121: part 3: invalidate on memory pressure

[ta...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 719ac4a87a226fadb250b0add0639107b38ab661
Author: Tim Armstrong <ta...@cloudera.com>
AuthorDate: Fri May 10 08:47:27 2019 -0700

    IMPALA-8121: part 3: invalidate on memory pressure
    
    Enable --invalidate_tables_on_memory_pressure=true
    on catalogd so that catalog can't hit out-of-memory.
    
    Testing:
    Ran core tests.
    
    Change-Id: I11d55ef0058abcf70f75b10ae9d89a0274859969
    Reviewed-on: http://gerrit.cloudera.org:8080/13302
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 docker/catalogd/Dockerfile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/docker/catalogd/Dockerfile b/docker/catalogd/Dockerfile
index 05e555d..b4d32b8 100644
--- a/docker/catalogd/Dockerfile
+++ b/docker/catalogd/Dockerfile
@@ -23,4 +23,5 @@ EXPOSE 25020
 
 ENTRYPOINT ["/opt/impala/bin/daemon_entrypoint.sh", "/opt/impala/bin/catalogd",\
      "-abort_on_config_error=false", "-state_store_host=statestored",\
-     "-catalog_topic_mode=minimal", "-hms_event_polling_interval_s=1"]
+     "-catalog_topic_mode=minimal", "-hms_event_polling_interval_s=1",\
+     "-invalidate_tables_on_memory_pressure=true"]

[impala] 01/02: Allow data cache to be enabled optionally when running tests

[ta...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 8f533e334c660dcdeaa4322b8755d7c0dce55f69
Author: Michael Ho <kw...@cloudera.com>
AuthorDate: Sun May 12 17:19:23 2019 -0700

    Allow data cache to be enabled optionally when running tests
    
    This change adds two environment variables ${DATA_CACHE_DIR}
    and ${DATA_CACHE_SIZE} which specify the root directory path
    of the data cache and the data cache size respectively. If both
    are set, Impala cluster will be started with data cache enabled
    when running E2E tests. When running with HDFS, it will also
    set the config option --always_use_data_cache to true to force
    use of the data cache all the time.
    
    Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b
    Reviewed-on: http://gerrit.cloudera.org:8080/13330
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/run-all-tests.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/bin/run-all-tests.sh b/bin/run-all-tests.sh
index c77d062..b74d03c 100755
--- a/bin/run-all-tests.sh
+++ b/bin/run-all-tests.sh
@@ -59,6 +59,10 @@ fi
 : ${RUN_TESTS_ARGS:=}
 # Extra args to pass to run-custom-cluster-tests.sh
 : ${RUN_CUSTOM_CLUSTER_TESTS_ARGS:=}
+# Data cache root directory location.
+: ${DATA_CACHE_DIR:=}
+# Data cache size.
+: ${DATA_CACHE_SIZE:=}
 if [[ "${TARGET_FILESYSTEM}" == "local" ]]; then
   # TODO: Remove abort_on_config_error flag from here and create-load-data.sh once
   # checkConfiguration() accepts the local filesystem (see IMPALA-1850).
@@ -69,6 +73,17 @@ else
   TEST_START_CLUSTER_ARGS="${TEST_START_CLUSTER_ARGS} --cluster_size=3"
 fi
 
+# Enable data cache if configured.
+if [[ -n "${DATA_CACHE_DIR}" && -n "${DATA_CACHE_SIZE}" ]]; then
+   TEST_START_CLUSTER_ARGS="${TEST_START_CLUSTER_ARGS} "`
+       `"--data_cache_dir=${DATA_CACHE_DIR} --data_cache_size=${DATA_CACHE_SIZE}"
+   # Force use of data cache for HDFS. Data cache is only enabled for remote reads.
+   if [[ "${TARGET_FILESYSTEM}" == "hdfs" ]]; then
+      TEST_START_CLUSTER_ARGS="${TEST_START_CLUSTER_ARGS} "`
+          `"--impalad_args=--always_use_data_cache=true"
+   fi
+fi
+
 if [[ "${ERASURE_CODING}" = true ]]; then
   # We do not run FE tests when erasure coding is enabled because planner tests
   # would fail.