You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/09/27 02:59:04 UTC
[impala] branch master updated (10c19b1a5 -> f6151b0aa)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 10c19b1a5 IMPALA-11511: Add build options for reducing binary sizes
new 49bffd236 IMPALA-11572: deflake test_mt_dop_skew_lpt part 2
new 0d7232d2f IMPALA-11438, IMPALA-11594: update test for non-HDFS
new f6151b0aa IMPALA-11585: Build quickstart_client with Ubuntu 20
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docker/CMakeLists.txt | 5 ++++-
docker/publish_images_to_apache.sh | 3 +--
docker/quickstart_client/Dockerfile | 3 ++-
tests/query_test/test_iceberg.py | 2 +-
tests/query_test/test_scanners.py | 2 ++
5 files changed, 10 insertions(+), 5 deletions(-)
[impala] 03/03: IMPALA-11585: Build quickstart_client with Ubuntu 20
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit f6151b0aa18cfcad50f8f63d5621f56db9fde6fe
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Wed Sep 14 11:30:15 2022 -0700
IMPALA-11585: Build quickstart_client with Ubuntu 20
Ubuntu 20.04 only provides the python3-pip package. Update building
quickstart_client to use python3-pip on Ubuntu 20.04.
Change-Id: Ife89b7db88dd58e96ba1b3e3972ca97204332dd4
Reviewed-on: http://gerrit.cloudera.org:8080/18984
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docker/CMakeLists.txt | 5 ++++-
docker/publish_images_to_apache.sh | 3 +--
docker/quickstart_client/Dockerfile | 3 ++-
3 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/docker/CMakeLists.txt b/docker/CMakeLists.txt
index 37396c67f..ad443e3f2 100644
--- a/docker/CMakeLists.txt
+++ b/docker/CMakeLists.txt
@@ -37,10 +37,13 @@ execute_process(COMMAND ${LSB_RELEASE_EXEC} -rs
if(${LSB_RELEASE_ID} STREQUAL "Ubuntu" AND ${LSB_RELEASE_VERSION} STREQUAL "16.04")
set(DISTRO_BASE_IMAGE "ubuntu:16.04")
+ set(PIP "python-pip")
elseif(${LSB_RELEASE_ID} STREQUAL "Ubuntu" AND ${LSB_RELEASE_VERSION} STREQUAL "18.04")
set(DISTRO_BASE_IMAGE "ubuntu:18.04")
+ set(PIP "python-pip")
elseif(${LSB_RELEASE_ID} STREQUAL "Ubuntu" AND ${LSB_RELEASE_VERSION} STREQUAL "20.04")
set(DISTRO_BASE_IMAGE "ubuntu:20.04")
+ set(PIP "python3-pip")
else()
set(DISTRO_BASE_IMAGE "UNSUPPORTED")
endif()
@@ -158,7 +161,7 @@ if (NOT ${DISTRO_BASE_IMAGE} STREQUAL "UNSUPPORTED")
# Supply the appropriate base image as an argument for the Dockerfile.
# Use tar with -h flag to assemble a tarball including all the symlinked files and
# directories in the build context.
- COMMAND tar cvh . -C ${quickstart_client_build_dir} . | ${DOCKER_BUILD} ${COMMON_DOCKER_BUILD_ARGS} --build-arg BASE_IMAGE=${DISTRO_BASE_IMAGE} -t ${QUICKSTART_CLIENT_IMAGE} -
+ COMMAND tar cvh . -C ${quickstart_client_build_dir} . | ${DOCKER_BUILD} ${COMMON_DOCKER_BUILD_ARGS} --build-arg BASE_IMAGE=${DISTRO_BASE_IMAGE} --build-arg PIP=${PIP} -t ${QUICKSTART_CLIENT_IMAGE} -
DEPENDS ${quickstart_client_build_dir}/Dockerfile ${quickstart_client_build_dir}/data-load-entrypoint.sh
COMMENT "Building quickstart client docker image."
VERBATIM
diff --git a/docker/publish_images_to_apache.sh b/docker/publish_images_to_apache.sh
index f0a2e64bc..42892f33b 100755
--- a/docker/publish_images_to_apache.sh
+++ b/docker/publish_images_to_apache.sh
@@ -20,8 +20,7 @@
set -euo pipefail
usage() {
- echo "publish_container_to_apache.sh -v <version string> [-d] [-r <repo>"
- echo " -d: if specified, upload debug images instead of release images"
+ echo "publish_container_to_apache.sh -v <version string> [-r <repo>]"
echo " -r: docker repository to upload to (defaults to apache/impala)"
echo " -v: version string to tag upload with, e.g. git hash or release version"
}
diff --git a/docker/quickstart_client/Dockerfile b/docker/quickstart_client/Dockerfile
index 96554f2c3..1f8c597c7 100644
--- a/docker/quickstart_client/Dockerfile
+++ b/docker/quickstart_client/Dockerfile
@@ -27,6 +27,7 @@ ARG VCS_REF
ARG VCS_TYPE
ARG VCS_URL
ARG VERSION
+ARG PIP=python-pip
# Install useful utilities. Set to non-interactive to avoid issues when installing tzdata.
ENV DEBIAN_FRONTEND=noninteractive
@@ -34,7 +35,7 @@ RUN apt-get update && \
apt-get install -y \
sudo netcat-openbsd less curl iproute2 vim iputils-ping \
libsasl2-dev libsasl2-2 libsasl2-modules libsasl2-modules-gssapi-mit \
- tzdata krb5-user python-pip && \
+ tzdata krb5-user ${PIP} && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
[impala] 01/03: IMPALA-11572: deflake test_mt_dop_skew_lpt part 2
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 49bffd236ca6caa4d86f988c3f7290b7dca0b2ff
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Fri Sep 23 15:41:02 2022 +0200
IMPALA-11572: deflake test_mt_dop_skew_lpt part 2
I suspect the test is still flaky because of parallel execution
of other tests that have high CPU usage, hence marking the test
for serial execution to make it more stable.
Also printing out the query profile to stdout when we observe a
skew.
Change-Id: I521685d683d51a52e54ad138f8466dd41c844f72
Reviewed-on: http://gerrit.cloudera.org:8080/19035
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/query_test/test_scanners.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/query_test/test_scanners.py b/tests/query_test/test_scanners.py
index f49ebbd53..29979929a 100644
--- a/tests/query_test/test_scanners.py
+++ b/tests/query_test/test_scanners.py
@@ -379,6 +379,7 @@ class TestHdfsScannerSkew(ImpalaTestSuite):
v.get_value('table_format').compression_codec == 'none')
@SkipIfLocal.multiple_impalad
+ @pytest.mark.execute_serially
def test_mt_dop_skew_lpt(self, vector, unique_database):
"""IMPALA-11539: Sanity check for MT scan nodes to make sure that the intra-node
skew is mitigated. For intra-node scan range assignment we are using dynamic
@@ -410,6 +411,7 @@ class TestHdfsScannerSkew(ImpalaTestSuite):
print "Intra-node bytes read ratio:", ratio
if ratio < SKEW_THRESHOLD:
count_skew += 1
+ print "Profile of skewed execution: ", profile
return count_skew
tbl_name = unique_database + ".lineitem_skew"
[impala] 02/03: IMPALA-11438, IMPALA-11594: update test for non-HDFS
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 0d7232d2f2c2105144a673ee7a413f77f12a995c
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Fri Sep 23 09:32:45 2022 -0700
IMPALA-11438, IMPALA-11594: update test for non-HDFS
Updates the new create_table_like_parquet test to work with S3 and Ozone
filesystem schemes in addition to HDFS.
Change-Id: Ibd8d4c6b96c3ed607556793e6b822944a879a1f8
Reviewed-on: http://gerrit.cloudera.org:8080/19037
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/query_test/test_iceberg.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/query_test/test_iceberg.py b/tests/query_test/test_iceberg.py
index 85f4d39e6..e0caccf63 100644
--- a/tests/query_test/test_iceberg.py
+++ b/tests/query_test/test_iceberg.py
@@ -804,7 +804,7 @@ class TestIcebergTable(IcebergTestSuite):
result_rows = results.stdout.strip().split('\n')
hdfs_file = None
for row in result_rows:
- if "hdfs://" in row:
+ if "://" in row:
hdfs_file = row.split('|')[1].lstrip()
break
assert hdfs_file