You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2019/03/27 15:50:29 UTC

[impala] branch master updated (36a67c4 -> 7079fc9)

This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.


    from 36a67c4  IMPALA-8308: [DOCS] Impala 3.2 Release
     new c63eebb  IMPALA-8362: Increase authorization test coverage with local catalog
     new eb90211  Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script
     new 1f5645c  Add missing sudo call to `service postgresql start`
     new fbe6cb4  IMPALA-8314 Set correct permissions for ~/.ssh files
     new 7cdeab7  IMPALA-8307 Use test -x to check for ntp-wait
     new 7079fc9  Use 'wget http://169.254.169.254/' to determine if we're running in aws

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 bin/bootstrap_system.sh                  | 16 +++++++---------
 testdata/cluster/admin                   | 17 ++++++++++-------
 tests/authorization/test_grant_revoke.py | 25 ++++++++++++++++++++-----
 3 files changed, 37 insertions(+), 21 deletions(-)


[impala] 06/06: Use 'wget http://169.254.169.254/' to determine if we're running in aws

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 7079fc99a91bc140fcb278d45c428ab90642f10c
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 18:18:53 2019 -0700

    Use 'wget http://169.254.169.254/' to determine if we're running in aws
    
    https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
    lists the above endpoint as the first choice.
    
    In my running instance 'dmidecode -s bios-version' prints out:
    1.0
    
    dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative
    here.
    
    I removed the 'grep' assertions since it's possible that images may have
    already configured ntp here.
    
    Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205
    Reviewed-on: http://gerrit.cloudera.org:8080/12727
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/bootstrap_system.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index cd7a891..98cba3f 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -237,11 +237,11 @@ redhat6 sudo service ntpd stop
 redhat7 notindocker sudo service ntpd stop
 sudo ntpdate us.pool.ntp.org
 # If on EC2, use Amazon's ntp servers
-if which dmidecode && { sudo dmidecode -s bios-version | grep amazon; }
+# EC2 nodes expose this IP address internally as a way to gather instance metadata.
+# The assumption is that only AWS nodes do this
+if wget -q -T 1 -t 1 -o /dev/null http://169.254.169.254/latest/dynamic/instance-identity
 then
   sudo sed -i 's/ubuntu\.pool/amazon\.pool/' /etc/ntp.conf
-  grep amazon /etc/ntp.conf
-  grep ubuntu /etc/ntp.conf
 fi
 # While it is nice to have ntpd running to keep the clock in sync, that does not work in a
 # --privileged docker container, and a non-privileged container cannot run ntpdate, which


[impala] 04/06: IMPALA-8314 Set correct permissions for ~/.ssh files

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit fbe6cb4e0189e1e7f00c5a1a501002e676e216ce
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 10:34:15 2019 -0700

    IMPALA-8314 Set correct permissions for ~/.ssh files
    
    Without this running bootstrap_system.sh resulted in:
    ++ chmod go-rwx /home/hacosta/.ssh
    ++ [[ -f /home/hacosta/.ssh/id_rsa ]]
    ++ ssh-keygen -t rsa -N '' -q -f /home/hacosta/.ssh/id_rsa
    ++ cat /home/hacosta/.ssh/id_rsa.pub
    ++ echo 'NoHostAuthenticationForLocalhost yes'
    ++ ssh localhost whoami
    Bad owner or permissions on /home/hacosta/.ssh/config
    
    Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087
    Reviewed-on: http://gerrit.cloudera.org:8080/12725
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/bootstrap_system.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index 574dd5d..cd7a891 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -299,8 +299,8 @@ if ! [[ -f ~/.ssh/id_rsa ]]
 then
   ssh-keygen -t rsa -N '' -q -f ~/.ssh/id_rsa
 fi
-cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 0600 ~/.ssh/authorized_keys
+echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config && chmod 0600 ~/.ssh/config
 ssh localhost whoami
 
 # Workarounds for HDFS networking issues: On the minicluster, tests that rely


[impala] 01/06: IMPALA-8362: Increase authorization test coverage with local catalog

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit c63eebb2e3e386e60c91a06d9cbe586fee92025b
Author: Fredy Wijaya <fw...@cloudera.com>
AuthorDate: Tue Mar 26 09:49:27 2019 -0700

    IMPALA-8362: Increase authorization test coverage with local catalog
    
    This patch adds authorization test that runs with local catalog by
    reusing the existing grant and revoke tests.
    
    Testing:
    - Added a new authorization test with local catalog
    - Ran all authorization tests
    
    Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a
    Reviewed-on: http://gerrit.cloudera.org:8080/12859
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 tests/authorization/test_grant_revoke.py | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/tests/authorization/test_grant_revoke.py b/tests/authorization/test_grant_revoke.py
index abcd7d7..0dcd423 100644
--- a/tests/authorization/test_grant_revoke.py
+++ b/tests/authorization/test_grant_revoke.py
@@ -109,20 +109,35 @@ class TestGrantRevoke(SentryCacheTestSuite):
       catalogd_args="--sentry_config={0} --sentry_catalog_polling_frequency_s={1}"
                     .format(SENTRY_CONFIG_FILE, SENTRY_LONG_POLLING_FREQUENCY_S),
       sentry_config=SENTRY_CONFIG_FILE)
-  def test_grant_revoke_with_sentry(self, vector):
+  def test_grant_revoke_with_catalog_v1(self):
+    """Tests grant and revoke using catalog v1."""
+    self._test_grant_revoke()
+
+  @pytest.mark.execute_serially
+  @SentryCacheTestSuite.with_args(
+    impalad_args="--server_name=server1 --use_local_catalog=true",
+    catalogd_args="--sentry_config={0} --sentry_catalog_polling_frequency_s={1} "
+                  "--use_local_catalog=true --catalog_topic_mode=minimal"
+                  .format(SENTRY_CONFIG_FILE, SENTRY_LONG_POLLING_FREQUENCY_S),
+    sentry_config=SENTRY_CONFIG_FILE)
+  def test_grant_revoke_with_local_catalog(self):
+    """Tests grant and revoke using local catalog mode."""
+    self._test_grant_revoke()
+
+  def _test_grant_revoke(self):
     """Tests grant and revoke for all objects. In these tests, we run tests twice. One
     with just using cache, hence the long sentry poll, and another one by ensuring
     refreshes happen from Sentry."""
     for refresh in [True, False]:
       for priv in ["all", "select"]:
         self._execute_with_grant_option_tests(
-            TestObject(TestObject.SERVER), priv, refresh)
+          TestObject(TestObject.SERVER), priv, refresh)
         self._execute_with_grant_option_tests(TestObject(
-            TestObject.DATABASE, "grant_rev_db"), priv, refresh)
+          TestObject.DATABASE, "grant_rev_db"), priv, refresh)
         self._execute_with_grant_option_tests(TestObject(
-            TestObject.TABLE, "grant_rev_db.tbl1"), priv, refresh)
+          TestObject.TABLE, "grant_rev_db.tbl1"), priv, refresh)
         self._execute_with_grant_option_tests(
-            TestObject(TestObject.VIEW, "grant_rev_db.tbl1"), priv, refresh)
+          TestObject(TestObject.VIEW, "grant_rev_db.tbl1"), priv, refresh)
 
   def _execute_with_grant_option_tests(self, test_obj, privilege, refresh_authorization):
     """


[impala] 02/06: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit eb9021119b332005ac409c32c064e47c0be6ec02
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 09:07:07 2019 -0700

    Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script
    
    This also changes the behavior to always use DEBIAN_FRONTEND=noninteractive
    even for interactive terminals.
    
    Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac
    Reviewed-on: http://gerrit.cloudera.org:8080/12723
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/bootstrap_system.sh | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index cb869ff..2e0f3d4 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -63,8 +63,6 @@ then
       exit 1
     fi
   done
-else
-  export DEBIAN_FRONTEND=noninteractive
 fi
 
 set -x
@@ -94,13 +92,13 @@ else
     echo "This script only supports Ubuntu or RedHat" >&2
     exit 1
   fi
-
   if ! [[ $DISTRIB_RELEASE = 16.04 ]]
   then
     echo "This script only supports 16.04 of Ubuntu" >&2
     exit 1
   fi
   UBUNTU=true
+  export DEBIAN_FRONTEND=noninteractive
 fi
 if grep docker /proc/1/cgroup; then
   IN_DOCKER=true


[impala] 05/06: IMPALA-8307 Use test -x to check for ntp-wait

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 7cdeab7f75bfcf98c15f8554586aa621cbe9b631
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 17:36:59 2019 -0700

    IMPALA-8307 Use test -x to check for ntp-wait
    
    Running ntp-wait --help can return 0 or 1 depending on the
    output of ntpq. For example:
    
    $ sudo ntp-wait --help 2> /dev/null; echo $?
    0
    $ sudo killall ntpd
    $ sudo ntp-wait --help 2> /dev/null; echo $?
    1
    
    This commit instead tests whether ntp-wait exists and is executable to
    determine if it's installed.
    
    Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c
    Reviewed-on: http://gerrit.cloudera.org:8080/12726
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 testdata/cluster/admin | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/testdata/cluster/admin b/testdata/cluster/admin
index a407d2c..acc44a5 100755
--- a/testdata/cluster/admin
+++ b/testdata/cluster/admin
@@ -335,13 +335,16 @@ function start_cluster {
         NTP_CANARY=pool.ntp.org
         if ! ping -c 1 -w 5 "${NTP_CANARY}" >/dev/null 2>/dev/null; then
           echo "WARNING: cannot reach ${NTP_CANARY}; ntp sync recommended for Kudu"
-        # ntp-wait is in /usr/sbin on both CentOS and Debian-based systems, but regular
-        # users don't always have /usr/sbin in $PATH. Use the absolute path.
-        elif ! /usr/sbin/ntp-wait --help >/dev/null 2>/dev/null; then
-          echo "WARNING: ntp-wait not installed; ntp sync recommended for Kudu"
-        elif ! /usr/sbin/ntp-wait -v; then
-          echo "ntp-wait failed; cannot start kudu"
-          return 1
+        else
+          # ntp-wait is in /usr/sbin on both CentOS and Debian-based systems, but regular
+          # users don't always have /usr/sbin in $PATH. Use the absolute path.
+          ntp_wait=$(command -v ntp-wait || echo /usr/sbin/ntp-wait)
+          if [[ ! -x $ntp_wait ]]; then
+            echo "WARNING: ntp-wait not installed; ntp sync recommended for Kudu"
+          elif ! $ntp_wait -v; then
+            echo "ntp-wait failed; cannot start kudu"
+            return 1
+          fi
         fi
       fi
       start $SERVICE


[impala] 03/06: Add missing sudo call to `service postgresql start`

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 1f5645c0d4027b326e90d40fe4705f8174c09ab2
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 09:48:04 2019 -0700

    Add missing sudo call to `service postgresql start`
    
    Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75
    Reviewed-on: http://gerrit.cloudera.org:8080/12724
    Reviewed-by: Joe McDonnell <jo...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/bootstrap_system.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index 2e0f3d4..574dd5d 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -278,7 +278,7 @@ redhat sudo sed -i -e 's,\(host.*\)ident,\1md5,' /var/lib/pgsql/data/pg_hba.conf
 
 ubuntu sudo service postgresql start
 redhat6 sudo service postgresql start
-redhat7 notindocker service postgresql start
+redhat7 notindocker sudo service postgresql start
 # Important to redirect pg_ctl to a logfile, lest it keep the stdout
 # file descriptor open, preventing the shell from exiting.
 redhat7 indocker sudo -u postgres PGDATA=/var/lib/pgsql/data bash -c \