You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2019/03/27 15:50:29 UTC
[impala] branch master updated (36a67c4 -> 7079fc9)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 36a67c4 IMPALA-8308: [DOCS] Impala 3.2 Release
new c63eebb IMPALA-8362: Increase authorization test coverage with local catalog
new eb90211 Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script
new 1f5645c Add missing sudo call to `service postgresql start`
new fbe6cb4 IMPALA-8314 Set correct permissions for ~/.ssh files
new 7cdeab7 IMPALA-8307 Use test -x to check for ntp-wait
new 7079fc9 Use 'wget http://169.254.169.254/' to determine if we're running in aws
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/bootstrap_system.sh | 16 +++++++---------
testdata/cluster/admin | 17 ++++++++++-------
tests/authorization/test_grant_revoke.py | 25 ++++++++++++++++++++-----
3 files changed, 37 insertions(+), 21 deletions(-)
[impala] 06/06: Use 'wget http://169.254.169.254/' to determine if
we're running in aws
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 7079fc99a91bc140fcb278d45c428ab90642f10c
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 18:18:53 2019 -0700
Use 'wget http://169.254.169.254/' to determine if we're running in aws
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
lists the above endpoint as the first choice.
In my running instance 'dmidecode -s bios-version' prints out:
1.0
dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative
here.
I removed the 'grep' assertions since it's possible that images may have
already configured ntp here.
Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205
Reviewed-on: http://gerrit.cloudera.org:8080/12727
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/bootstrap_system.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index cd7a891..98cba3f 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -237,11 +237,11 @@ redhat6 sudo service ntpd stop
redhat7 notindocker sudo service ntpd stop
sudo ntpdate us.pool.ntp.org
# If on EC2, use Amazon's ntp servers
-if which dmidecode && { sudo dmidecode -s bios-version | grep amazon; }
+# EC2 nodes expose this IP address internally as a way to gather instance metadata.
+# The assumption is that only AWS nodes do this
+if wget -q -T 1 -t 1 -o /dev/null http://169.254.169.254/latest/dynamic/instance-identity
then
sudo sed -i 's/ubuntu\.pool/amazon\.pool/' /etc/ntp.conf
- grep amazon /etc/ntp.conf
- grep ubuntu /etc/ntp.conf
fi
# While it is nice to have ntpd running to keep the clock in sync, that does not work in a
# --privileged docker container, and a non-privileged container cannot run ntpdate, which
[impala] 04/06: IMPALA-8314 Set correct permissions for ~/.ssh files
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit fbe6cb4e0189e1e7f00c5a1a501002e676e216ce
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 10:34:15 2019 -0700
IMPALA-8314 Set correct permissions for ~/.ssh files
Without this running bootstrap_system.sh resulted in:
++ chmod go-rwx /home/hacosta/.ssh
++ [[ -f /home/hacosta/.ssh/id_rsa ]]
++ ssh-keygen -t rsa -N '' -q -f /home/hacosta/.ssh/id_rsa
++ cat /home/hacosta/.ssh/id_rsa.pub
++ echo 'NoHostAuthenticationForLocalhost yes'
++ ssh localhost whoami
Bad owner or permissions on /home/hacosta/.ssh/config
Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087
Reviewed-on: http://gerrit.cloudera.org:8080/12725
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/bootstrap_system.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index 574dd5d..cd7a891 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -299,8 +299,8 @@ if ! [[ -f ~/.ssh/id_rsa ]]
then
ssh-keygen -t rsa -N '' -q -f ~/.ssh/id_rsa
fi
-cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 0600 ~/.ssh/authorized_keys
+echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config && chmod 0600 ~/.ssh/config
ssh localhost whoami
# Workarounds for HDFS networking issues: On the minicluster, tests that rely
[impala] 01/06: IMPALA-8362: Increase authorization test coverage
with local catalog
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit c63eebb2e3e386e60c91a06d9cbe586fee92025b
Author: Fredy Wijaya <fw...@cloudera.com>
AuthorDate: Tue Mar 26 09:49:27 2019 -0700
IMPALA-8362: Increase authorization test coverage with local catalog
This patch adds authorization test that runs with local catalog by
reusing the existing grant and revoke tests.
Testing:
- Added a new authorization test with local catalog
- Ran all authorization tests
Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a
Reviewed-on: http://gerrit.cloudera.org:8080/12859
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/authorization/test_grant_revoke.py | 25 ++++++++++++++++++++-----
1 file changed, 20 insertions(+), 5 deletions(-)
diff --git a/tests/authorization/test_grant_revoke.py b/tests/authorization/test_grant_revoke.py
index abcd7d7..0dcd423 100644
--- a/tests/authorization/test_grant_revoke.py
+++ b/tests/authorization/test_grant_revoke.py
@@ -109,20 +109,35 @@ class TestGrantRevoke(SentryCacheTestSuite):
catalogd_args="--sentry_config={0} --sentry_catalog_polling_frequency_s={1}"
.format(SENTRY_CONFIG_FILE, SENTRY_LONG_POLLING_FREQUENCY_S),
sentry_config=SENTRY_CONFIG_FILE)
- def test_grant_revoke_with_sentry(self, vector):
+ def test_grant_revoke_with_catalog_v1(self):
+ """Tests grant and revoke using catalog v1."""
+ self._test_grant_revoke()
+
+ @pytest.mark.execute_serially
+ @SentryCacheTestSuite.with_args(
+ impalad_args="--server_name=server1 --use_local_catalog=true",
+ catalogd_args="--sentry_config={0} --sentry_catalog_polling_frequency_s={1} "
+ "--use_local_catalog=true --catalog_topic_mode=minimal"
+ .format(SENTRY_CONFIG_FILE, SENTRY_LONG_POLLING_FREQUENCY_S),
+ sentry_config=SENTRY_CONFIG_FILE)
+ def test_grant_revoke_with_local_catalog(self):
+ """Tests grant and revoke using local catalog mode."""
+ self._test_grant_revoke()
+
+ def _test_grant_revoke(self):
"""Tests grant and revoke for all objects. In these tests, we run tests twice. One
with just using cache, hence the long sentry poll, and another one by ensuring
refreshes happen from Sentry."""
for refresh in [True, False]:
for priv in ["all", "select"]:
self._execute_with_grant_option_tests(
- TestObject(TestObject.SERVER), priv, refresh)
+ TestObject(TestObject.SERVER), priv, refresh)
self._execute_with_grant_option_tests(TestObject(
- TestObject.DATABASE, "grant_rev_db"), priv, refresh)
+ TestObject.DATABASE, "grant_rev_db"), priv, refresh)
self._execute_with_grant_option_tests(TestObject(
- TestObject.TABLE, "grant_rev_db.tbl1"), priv, refresh)
+ TestObject.TABLE, "grant_rev_db.tbl1"), priv, refresh)
self._execute_with_grant_option_tests(
- TestObject(TestObject.VIEW, "grant_rev_db.tbl1"), priv, refresh)
+ TestObject(TestObject.VIEW, "grant_rev_db.tbl1"), priv, refresh)
def _execute_with_grant_option_tests(self, test_obj, privilege, refresh_authorization):
"""
[impala] 02/06: Move DEBIAN_FRONTEND=noninteractive variable to the
ubuntu section of the script
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit eb9021119b332005ac409c32c064e47c0be6ec02
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 09:07:07 2019 -0700
Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script
This also changes the behavior to always use DEBIAN_FRONTEND=noninteractive
even for interactive terminals.
Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac
Reviewed-on: http://gerrit.cloudera.org:8080/12723
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/bootstrap_system.sh | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index cb869ff..2e0f3d4 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -63,8 +63,6 @@ then
exit 1
fi
done
-else
- export DEBIAN_FRONTEND=noninteractive
fi
set -x
@@ -94,13 +92,13 @@ else
echo "This script only supports Ubuntu or RedHat" >&2
exit 1
fi
-
if ! [[ $DISTRIB_RELEASE = 16.04 ]]
then
echo "This script only supports 16.04 of Ubuntu" >&2
exit 1
fi
UBUNTU=true
+ export DEBIAN_FRONTEND=noninteractive
fi
if grep docker /proc/1/cgroup; then
IN_DOCKER=true
[impala] 05/06: IMPALA-8307 Use test -x to check for ntp-wait
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 7cdeab7f75bfcf98c15f8554586aa621cbe9b631
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 17:36:59 2019 -0700
IMPALA-8307 Use test -x to check for ntp-wait
Running ntp-wait --help can return 0 or 1 depending on the
output of ntpq. For example:
$ sudo ntp-wait --help 2> /dev/null; echo $?
0
$ sudo killall ntpd
$ sudo ntp-wait --help 2> /dev/null; echo $?
1
This commit instead tests whether ntp-wait exists and is executable to
determine if it's installed.
Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c
Reviewed-on: http://gerrit.cloudera.org:8080/12726
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/cluster/admin | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/testdata/cluster/admin b/testdata/cluster/admin
index a407d2c..acc44a5 100755
--- a/testdata/cluster/admin
+++ b/testdata/cluster/admin
@@ -335,13 +335,16 @@ function start_cluster {
NTP_CANARY=pool.ntp.org
if ! ping -c 1 -w 5 "${NTP_CANARY}" >/dev/null 2>/dev/null; then
echo "WARNING: cannot reach ${NTP_CANARY}; ntp sync recommended for Kudu"
- # ntp-wait is in /usr/sbin on both CentOS and Debian-based systems, but regular
- # users don't always have /usr/sbin in $PATH. Use the absolute path.
- elif ! /usr/sbin/ntp-wait --help >/dev/null 2>/dev/null; then
- echo "WARNING: ntp-wait not installed; ntp sync recommended for Kudu"
- elif ! /usr/sbin/ntp-wait -v; then
- echo "ntp-wait failed; cannot start kudu"
- return 1
+ else
+ # ntp-wait is in /usr/sbin on both CentOS and Debian-based systems, but regular
+ # users don't always have /usr/sbin in $PATH. Use the absolute path.
+ ntp_wait=$(command -v ntp-wait || echo /usr/sbin/ntp-wait)
+ if [[ ! -x $ntp_wait ]]; then
+ echo "WARNING: ntp-wait not installed; ntp sync recommended for Kudu"
+ elif ! $ntp_wait -v; then
+ echo "ntp-wait failed; cannot start kudu"
+ return 1
+ fi
fi
fi
start $SERVICE
[impala] 03/06: Add missing sudo call to `service postgresql start`
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 1f5645c0d4027b326e90d40fe4705f8174c09ab2
Author: Hector Acosta <he...@cloudera.com>
AuthorDate: Mon Mar 11 09:48:04 2019 -0700
Add missing sudo call to `service postgresql start`
Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75
Reviewed-on: http://gerrit.cloudera.org:8080/12724
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/bootstrap_system.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index 2e0f3d4..574dd5d 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -278,7 +278,7 @@ redhat sudo sed -i -e 's,\(host.*\)ident,\1md5,' /var/lib/pgsql/data/pg_hba.conf
ubuntu sudo service postgresql start
redhat6 sudo service postgresql start
-redhat7 notindocker service postgresql start
+redhat7 notindocker sudo service postgresql start
# Important to redirect pg_ctl to a logfile, lest it keep the stdout
# file descriptor open, preventing the shell from exiting.
redhat7 indocker sudo -u postgres PGDATA=/var/lib/pgsql/data bash -c \