You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2018/11/20 00:00:47 UTC
[1/3] impala git commit: IMPALA-5031: method calls on NULL are not
UBSAN-clean
Repository: impala
Updated Branches:
refs/heads/master 93a0ce857 -> 8f151d0e6
IMPALA-5031: method calls on NULL are not UBSAN-clean
According to [expr.post] in the C++14 standard, a call to a member
function like a->b() is interpreted as (a->b)(). In other words, the
dereferencing is done separately from the call. This makes calling
member functions on nullptr undefined behavior, since the dereference
invokes undefined behavior.
Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115
Reviewed-on: http://gerrit.cloudera.org:8080/11950
Reviewed-by: Jim Apple <jb...@apache.org>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/48d1d2d7
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/48d1d2d7
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/48d1d2d7
Branch: refs/heads/master
Commit: 48d1d2d7421d668f8dd5142b05395e2b163f621b
Parents: 93a0ce8
Author: Jim Apple <jb...@apache.org>
Authored: Sat Nov 17 19:08:21 2018 -0800
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Mon Nov 19 23:11:40 2018 +0000
----------------------------------------------------------------------
be/src/udf/udf.cc | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/48d1d2d7/be/src/udf/udf.cc
----------------------------------------------------------------------
diff --git a/be/src/udf/udf.cc b/be/src/udf/udf.cc
index 5edfec3..0688805 100644
--- a/be/src/udf/udf.cc
+++ b/be/src/udf/udf.cc
@@ -78,9 +78,7 @@ class FreePool {
class MemPool {
public:
- uint8_t* Allocate(int byte_size) {
- // TODO: this function is called with this == nullptr from UdaTestHarness. This works
- // for now because MemPool has no members or virtual functions.
+ static uint8_t* Allocate(int byte_size) {
return reinterpret_cast<uint8_t*>(malloc(byte_size));
}
};
[2/3] impala git commit: IMPALA-7871: Don't load Hive builtins
Posted by jo...@apache.org.
IMPALA-7871: Don't load Hive builtins
Dataload has a step of "Loading Hive builtins" that
loads a bunch of jars into HDFS/S3/etc. Despite
its name, nothing seems to be using these.
Dataload and core tests succeed without this step.
This removes the Hive builtins step and associated
scripts.
Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844
Reviewed-on: http://gerrit.cloudera.org:8080/11944
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/70fbd1df
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/70fbd1df
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/70fbd1df
Branch: refs/heads/master
Commit: 70fbd1df446f4830b55d684d82c851867b9c1444
Parents: 48d1d2d
Author: Joe McDonnell <jo...@cloudera.com>
Authored: Fri Nov 16 12:14:27 2018 -0800
Committer: Joe McDonnell <jo...@cloudera.com>
Committed: Mon Nov 19 23:33:20 2018 +0000
----------------------------------------------------------------------
testdata/bin/create-load-data.sh | 4 --
testdata/bin/load-hive-builtins.sh | 69 -----------------------
testdata/bin/load-test-warehouse-snapshot.sh | 7 ---
3 files changed, 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/70fbd1df/testdata/bin/create-load-data.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/create-load-data.sh b/testdata/bin/create-load-data.sh
index 74ae248..2a67bd4 100755
--- a/testdata/bin/create-load-data.sh
+++ b/testdata/bin/create-load-data.sh
@@ -98,10 +98,6 @@ do
done
if [[ $SKIP_METADATA_LOAD -eq 0 && "$SNAPSHOT_FILE" = "" ]]; then
- if [[ -z "$REMOTE_LOAD" ]]; then
- run-step "Loading Hive Builtins" load-hive-builtins.log \
- ${IMPALA_HOME}/testdata/bin/load-hive-builtins.sh
- fi
run-step "Generating HBase data" create-hbase.log \
${IMPALA_HOME}/testdata/bin/create-hbase.sh
run-step "Creating /test-warehouse HDFS directory" create-test-warehouse-dir.log \
http://git-wip-us.apache.org/repos/asf/impala/blob/70fbd1df/testdata/bin/load-hive-builtins.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/load-hive-builtins.sh b/testdata/bin/load-hive-builtins.sh
deleted file mode 100755
index 55cc845..0000000
--- a/testdata/bin/load-hive-builtins.sh
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -euo pipefail
-. $IMPALA_HOME/bin/report_build_error.sh
-setup_report_build_error
-
-. ${IMPALA_HOME}/bin/impala-config.sh > /dev/null 2>&1
-
-# TODO: remove this once we understand why Hive looks in HDFS for many of its jars
-
-# Remove all directories in one command for efficiency
-${HADOOP_HOME}/bin/hadoop fs -rm -skipTrash -r -f ${FILESYSTEM_PREFIX}${HIVE_HOME}/lib/ \
- ${FILESYSTEM_PREFIX}${HBASE_HOME}/lib/ \
- ${FILESYSTEM_PREFIX}${HADOOP_HOME}/share/hadoop/common/ \
- ${FILESYSTEM_PREFIX}${HADOOP_HOME}/share/hadoop/mapreduce/ \
- ${FILESYSTEM_PREFIX}${HADOOP_HOME}/share/hadoop/tools/lib \
- ${FILESYSTEM_PREFIX}${HADOOP_LZO}/build \
- ${FILESYSTEM_PREFIX}${SENTRY_HOME}/lib/ \
- ${FILESYSTEM_PREFIX}${IMPALA_HOME}/thirdparty/postgresql-jdbc/
-
-TMP_DIR=$(mktemp -d)
-
-# Create the directory structure to copy over
-mkdir -p ${TMP_DIR}/${HIVE_HOME}/lib \
- ${TMP_DIR}/${HBASE_HOME}/lib \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/common/lib \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/mapreduce \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/tools/lib \
- ${TMP_DIR}/${HADOOP_LZO}/build \
- ${TMP_DIR}/${SENTRY_HOME}/lib \
- ${TMP_DIR}/${IMPALA_HOME}/thirdparty/postgresql-jdbc/
-
-# Add symbolic links to files in the appropriate places
-ln -s ${HIVE_HOME}/lib/*.jar ${TMP_DIR}/${HIVE_HOME}/lib
-ln -s ${HBASE_HOME}/lib/*.jar ${TMP_DIR}/${HBASE_HOME}/lib
-ln -s ${HADOOP_HOME}/share/hadoop/common/*.jar \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/common
-ln -s ${HADOOP_HOME}/share/hadoop/common/lib/*.jar \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/common/lib
-ln -s ${HADOOP_HOME}/share/hadoop/mapreduce/*.jar \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/mapreduce
-ln -s ${HADOOP_HOME}/share/hadoop/tools/lib/*.jar \
- ${TMP_DIR}/${HADOOP_HOME}/share/hadoop/tools/lib
-ln -s ${HADOOP_LZO}/build/hadoop-lzo*.jar ${TMP_DIR}/${HADOOP_LZO}/build
-ln -s ${SENTRY_HOME}/lib/*.jar ${TMP_DIR}/${SENTRY_HOME}/lib
-# This is the only item that uses a different path
-# TODO: why is this path different?
-ln -s ${POSTGRES_JDBC_DRIVER} ${TMP_DIR}/${IMPALA_HOME}/thirdparty/postgresql-jdbc
-
-${HADOOP_HOME}/bin/hadoop fs -put ${TMP_DIR}/* ${FILESYSTEM_PREFIX}/
-
-rm -r ${TMP_DIR}
http://git-wip-us.apache.org/repos/asf/impala/blob/70fbd1df/testdata/bin/load-test-warehouse-snapshot.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/load-test-warehouse-snapshot.sh b/testdata/bin/load-test-warehouse-snapshot.sh
index da5fb07..cfec558 100755
--- a/testdata/bin/load-test-warehouse-snapshot.sh
+++ b/testdata/bin/load-test-warehouse-snapshot.sh
@@ -102,13 +102,6 @@ if [ ! -f ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/githash.txt ]; then
exit 1
fi
-
-# Hive builtins are already present on a pre-setup CM managed cluster.
-if [[ -z "$REMOTE_LOAD" ]]; then
- echo "Loading hive builtins"
- ${IMPALA_HOME}/testdata/bin/load-hive-builtins.sh
-fi
-
echo "Copying data to ${TARGET_FILESYSTEM}"
if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
# hive does not yet work well with s3, so we won't need hive builtins.
[3/3] impala git commit: IMPALA-5031: Zero-length arrays are
undefined behavior
Posted by jo...@apache.org.
IMPALA-5031: Zero-length arrays are undefined behavior
This patch prevents the creation of a zero-length array. This is
illegal under paragraph 5 of §6.7.5.2 ("Array declarators") of the C99
standard, which reads:
If the size is an expression that is not an integer constant
expression: if it occurs in a declaration at function prototype
scope, it is treated as if it were replaced by *; otherwise, each
time it is evaluated it shall have a value greater than zero.
Variable-length arrays are not part of C++14, but they are a common
compiler extension and are available in both clang++ and g++. That the
semantics of missing features are the same is implied by [intro.scope]
in the C++14 standard, which reads:
C++ is a general purpose programming language based on the C
programming language as described in ISO/IEC 9899:1999 Programming
languages — C (hereinafter referred to as the C standard).
Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96
Reviewed-on: http://gerrit.cloudera.org:8080/11811
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/8f151d0e
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/8f151d0e
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/8f151d0e
Branch: refs/heads/master
Commit: 8f151d0e63aa5f3548e283fadcaf801456cb3354
Parents: 70fbd1d
Author: Jim Apple <jb...@apache.org>
Authored: Sat Oct 27 16:25:39 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Mon Nov 19 23:41:23 2018 +0000
----------------------------------------------------------------------
be/src/runtime/row-batch-serialize-test.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/8f151d0e/be/src/runtime/row-batch-serialize-test.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/row-batch-serialize-test.cc b/be/src/runtime/row-batch-serialize-test.cc
index e765025..4ea81a6 100644
--- a/be/src/runtime/row-batch-serialize-test.cc
+++ b/be/src/runtime/row-batch-serialize-test.cc
@@ -240,7 +240,7 @@ class RowBatchSerializeTest : public testing::Test {
"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
"abcdefghijklmnopqrstuvwxyz";
int len = rand() % (MAX_STRING_LEN + 1);
- char buf[len];
+ char buf[MAX_STRING_LEN];
for (int i = 0; i < len; ++i) {
buf[i] = chars[rand() % (sizeof(chars) - 1)];
}