You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2024/01/31 00:23:01 UTC
(impala) branch master updated (eaa35b025 -> 46f043132)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from eaa35b025 IMPALA-12463: Batch non-consecutive table events in the event processor
new f8302d903 IMPALA-12767: Upgrade Guava to 32.0.1 due to CVE-2023-2976
new 18a77cd3b IMPALA-12762: Fix cmake error in package building
new ab445195b IMPALA-12756: [DOCS] Unicode column name support documentation
new 46f043132 IMPALA-12763: Union with string struct crashes in ASAN
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/json/CMakeLists.txt | 4 ++++
bin/impala-config.sh | 2 +-
bin/jenkins/build-all-flag-combinations.sh | 4 ++--
docs/topics/impala_identifiers.xml | 10 ++++++----
.../java/org/apache/impala/analysis/Analyzer.java | 8 +++++---
.../main/java/org/apache/impala/catalog/Type.java | 22 ++++++++++++++++++++++
.../apache/impala/analysis/AnalyzeStmtsTest.java | 10 ++++++++--
.../QueryTest/mixed-collections-and-structs.test | 6 ++++++
.../QueryTest/nested-array-in-select-list.test | 2 +-
.../QueryTest/nested-map-in-select-list.test | 2 +-
.../queries/QueryTest/struct-in-select-list.test | 2 +-
11 files changed, 57 insertions(+), 15 deletions(-)
(impala) 02/04: IMPALA-12762: Fix cmake error in package building
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 18a77cd3bcafa7d650177ad8b6ad1db8cf21a21d
Author: zhangyifan27 <ch...@163.com>
AuthorDate: Mon Jan 29 17:44:20 2024 +0800
IMPALA-12762: Fix cmake error in package building
This patch adds extra processing of option 'BUILD_WITH_NO_TESTS' in
be/src/exec/json/CMakeLists.txt, so test targets will not be generated
by the CMake when building Impala with -package and -notests.
Testing:
- Run './buildall.sh -noclean -notests -package' with no error
Change-Id: Ice0cbb0671d915f997fa74217521a82be164ae57
Reviewed-on: http://gerrit.cloudera.org:8080/20965
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/exec/json/CMakeLists.txt | 4 ++++
bin/jenkins/build-all-flag-combinations.sh | 4 ++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/be/src/exec/json/CMakeLists.txt b/be/src/exec/json/CMakeLists.txt
index e654ad9f4..5ad000c5a 100644
--- a/be/src/exec/json/CMakeLists.txt
+++ b/be/src/exec/json/CMakeLists.txt
@@ -28,6 +28,10 @@ add_library(ExecJson
add_dependencies(ExecJson gen-deps)
+if (BUILD_WITH_NO_TESTS)
+ return()
+endif()
+
add_library(ExecJsonTests STATIC
json-parser-test.cc
)
diff --git a/bin/jenkins/build-all-flag-combinations.sh b/bin/jenkins/build-all-flag-combinations.sh
index e54ab396b..1143f706b 100755
--- a/bin/jenkins/build-all-flag-combinations.sh
+++ b/bin/jenkins/build-all-flag-combinations.sh
@@ -39,7 +39,7 @@ CONFIGS=(
# Test gcc builds with and without -so:
"-skiptests -noclean"
"-skiptests -noclean -release"
- "-skiptests -noclean -release -package"
+ "-notests -noclean -release -package"
"-skiptests -noclean -release -so -ninja"
# clang sanitizer builds:
"-skiptests -noclean -asan"
@@ -47,7 +47,7 @@ CONFIGS=(
"-skiptests -noclean -ubsan -so -ninja"
# USE_APACHE_HIVE=true build:
"-skiptests -noclean -use_apache_hive"
- "-skiptests -noclean -use_apache_hive -package"
+ "-notests -noclean -use_apache_hive -package"
)
FAILED=""
(impala) 01/04: IMPALA-12767: Upgrade Guava to 32.0.1 due to CVE-2023-2976
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit f8302d90369f2c0fd912963eb764063f4a9a41e2
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Mon Jan 29 10:14:00 2024 -0800
IMPALA-12767: Upgrade Guava to 32.0.1 due to CVE-2023-2976
This patch upgrade Guava version from 31.1-jre to 32.0.1-jre to address
CVE-2023-2976.
Testing:
- Run and pass full build
./buildall.sh -skiptests -notests
Change-Id: Id932ed32200fba4f24b4fdd108546ac4494fc3e8
Reviewed-on: http://gerrit.cloudera.org:8080/20972
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
---
bin/impala-config.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 8aa37f036..0886006bd 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -276,7 +276,7 @@ export APACHE_OZONE_VERSION=1.3.0
export IMPALA_BOUNCY_CASTLE_VERSION=1.68
export IMPALA_COS_VERSION=3.1.0-8.0.8
export IMPALA_DERBY_VERSION=10.14.2.0
-export IMPALA_GUAVA_VERSION=31.1-jre
+export IMPALA_GUAVA_VERSION=32.0.1-jre
export IMPALA_HUDI_VERSION=0.5.0-incubating
export IMPALA_HTTP_CORE_VERSION=4.4.14
export IMPALA_JACKSON_DATABIND_VERSION=2.15.3
(impala) 04/04: IMPALA-12763: Union with string struct crashes in ASAN
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 46f04313212952ae2e8f432cb622457918bae6cd
Author: Daniel Becker <da...@cloudera.com>
AuthorDate: Mon Jan 29 16:11:31 2024 +0100
IMPALA-12763: Union with string struct crashes in ASAN
In ASAN builds, if we UNION ALL an array containing a struct of a string
with itself, Impala crashes. This is how to reproduce it:
In Hive:
create table su (arr ARRAY<STRUCT<s: STRING>>) stored as parquet;
insert into su values (array(named_struct("s", "A")));
In Impala:
select 1, arr from su
union all select 2, arr from su;
The ASAN error message indicates a heap-use-after-free.
Normally, UNIONs of structs are not supported yet (see IMPALA-10752),
but if the struct is inside an array it is allowed now. This was
probably not intentional and it leads to the above error, so this change
disables structs in unions completely, including embedded structs.
Testing:
- adjusted existing tests
- added a query that tests that types with embedded structs are not
allowed in a UNION statement, in mixed-collections-and-structs.test
Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Reviewed-on: http://gerrit.cloudera.org:8080/20970
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../java/org/apache/impala/analysis/Analyzer.java | 8 +++++---
.../main/java/org/apache/impala/catalog/Type.java | 22 ++++++++++++++++++++++
.../apache/impala/analysis/AnalyzeStmtsTest.java | 10 ++++++++--
.../QueryTest/mixed-collections-and-structs.test | 6 ++++++
.../QueryTest/nested-array-in-select-list.test | 2 +-
.../QueryTest/nested-map-in-select-list.test | 2 +-
.../queries/QueryTest/struct-in-select-list.test | 2 +-
7 files changed, 44 insertions(+), 8 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index f6ce7b2bd..600cc9a8f 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -48,6 +48,7 @@ import org.apache.impala.authorization.PrivilegeRequest;
import org.apache.impala.authorization.PrivilegeRequestBuilder;
import org.apache.impala.authorization.TableMask;
import org.apache.impala.authorization.User;
+import org.apache.impala.catalog.ArrayType;
import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.DatabaseNotFoundException;
import org.apache.impala.catalog.FeCatalog;
@@ -62,6 +63,7 @@ import org.apache.impala.catalog.FeTable;
import org.apache.impala.catalog.FeView;
import org.apache.impala.catalog.IcebergTimeTravelTable;
import org.apache.impala.catalog.KuduTable;
+import org.apache.impala.catalog.MapType;
import org.apache.impala.catalog.MaterializedViewHdfsTable;
import org.apache.impala.catalog.ScalarType;
import org.apache.impala.catalog.StructField;
@@ -3400,10 +3402,10 @@ public class Analyzer {
// Initialize with type of i-th expr in first list.
Type compatibleType = firstList.get(i).getType();
if (firstList.get(i) instanceof SlotRef &&
- compatibleType.isStructType()) {
+ compatibleType.containsStruct()) {
throw new AnalysisException(String.format(
- "Set operations don't support STRUCT type. %s in %s", compatibleType.toSql(),
- firstList.get(i).toSql()));
+ "Set operations don't support STRUCT types or types containing " +
+ "STRUCT types. %s in %s.", compatibleType.toSql(), firstList.get(i).toSql()));
}
widestExprs.add(firstList.get(i));
diff --git a/fe/src/main/java/org/apache/impala/catalog/Type.java b/fe/src/main/java/org/apache/impala/catalog/Type.java
index f0246d2f9..98c051da8 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Type.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Type.java
@@ -243,6 +243,28 @@ public abstract class Type {
return this instanceof CollectionStructType;
}
+ /**
+ * Returns true if this type
+ * - is a struct type or
+ * - contains a struct type (recursively); for example
+ * ARRAY<STRUCT<i: INT>>.
+ */
+ public boolean containsStruct() {
+ if (isStructType()) return true;
+
+ if (isArrayType()) {
+ ArrayType arrayType = (ArrayType) this;
+ return arrayType.getItemType().containsStruct();
+ } else if (isMapType()) {
+ MapType mapType = (MapType) this;
+ return mapType.getKeyType().containsStruct() ||
+ mapType.getValueType().containsStruct();
+ }
+
+ return false;
+ }
+
+
/**
* Returns true if this type
* - is a collection type or
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
index fd0dbc043..6da0a5b45 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
@@ -1016,7 +1016,7 @@ public class AnalyzeStmtsTest extends AnalyzerTest {
// Empty star expansion, but non empty result exprs.
AnalyzesOk("select 1, * from only_complex_types");
- // Struct in select list works only if codegen is OFF.
+ // Struct in select list.
AnalysisContext ctx = createAnalysisCtx();
AnalyzesOk("select alltypes from functional_orc_def.complextypes_structs", ctx);
AnalyzesOk("select int_array_col from functional.allcomplextypes");
@@ -1030,7 +1030,13 @@ public class AnalyzeStmtsTest extends AnalyzerTest {
"collection 'int_array_col' of type 'ARRAY<INT>'");
AnalysisError("select tiny_struct from functional_orc_def.complextypes_structs " +
"union all select tiny_struct from functional_orc_def.complextypes_structs", ctx,
- "Set operations don't support STRUCT type. STRUCT<b:BOOLEAN> in tiny_struct");
+ "Set operations don't support STRUCT types or types containing STRUCT types." +
+ " STRUCT<b:BOOLEAN> in tiny_struct");
+ AnalysisError("select all_mix from functional_parquet.collection_struct_mix " +
+ "union all select all_mix from functional_parquet.collection_struct_mix", ctx,
+ "Set operations don't support STRUCT types or types containing STRUCT types. " +
+ "MAP<INT,STRUCT<big:STRUCT<arr:ARRAY<STRUCT<inner_arr:ARRAY<ARRAY<INT>>," +
+ "m:TIMESTAMP>>,n:INT>,small:STRUCT<str:STRING,i:INT>>> in all_mix.");
AnalyzesOk("select 1 from " +
"(select int_array_col from functional.allcomplextypes) v");
AnalyzesOk("select int_array_col from " +
diff --git a/testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test b/testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
index 56175127a..732aac87c 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
@@ -549,3 +549,9 @@ from collection_struct_mix, collection_struct_mix.arr_contains_nested_struct arr
---- TYPES
INT,STRING,SMALLINT
====
+---- QUERY
+# Union of types containing structs are not allowed.
+select all_mix from collection_struct_mix union all select all_mix from collection_struct_mix
+---- CATCH
+AnalysisException: Set operations don't support STRUCT types or types containing STRUCT types. MAP<INT,STRUCT<big:STRUCT<arr:ARRAY<STRUCT<inner_arr:ARRAY<ARRAY<INT>>,m:TIMESTAMP>>,n:INT>,small:STRUCT<str:STRING,i:INT>>> in all_mix.
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test b/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
index 4d4abc5ab..13dae41ea 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
@@ -133,7 +133,7 @@ tinyint,string,string
select 1, struct_contains_arr, struct_contains_nested_arr, all_mix from collection_struct_mix
union all select 2, struct_contains_arr, struct_contains_nested_arr, all_mix from collection_struct_mix
---- CATCH
-AnalysisException: Set operations don't support STRUCT type. STRUCT<arr:ARRAY<INT>> in struct_contains_arr
+AnalysisException: Set operations don't support STRUCT types or types containing STRUCT types. STRUCT<arr:ARRAY<INT>> in struct_contains_arr.
====
---- QUERY
select 1 from (select int_array from complextypestbl) s
diff --git a/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test b/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
index aedebd289..e21747c08 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
@@ -193,7 +193,7 @@ tinyint,string,string
select 1, struct_contains_map, all_mix from collection_struct_mix
union all select 2, struct_contains_map, all_mix from collection_struct_mix
---- CATCH
-AnalysisException: Set operations don't support STRUCT type. STRUCT<m:MAP<INT,STRING>> in struct_contains_map
+AnalysisException: Set operations don't support STRUCT types or types containing STRUCT types. STRUCT<m:MAP<INT,STRING>> in struct_contains_map.
====
---- QUERY
select 1 from (select int_map from complextypestbl) s
diff --git a/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test b/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
index 082892649..546d1c527 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
@@ -603,7 +603,7 @@ select id, tiny_struct from complextypes_structs
union all
select id, tiny_struct from complextypes_structs;
---- CATCH
-AnalysisException: Set operations don't support STRUCT type. STRUCT<b:BOOLEAN> in tiny_struct
+AnalysisException: Set operations don't support STRUCT types or types containing STRUCT types. STRUCT<b:BOOLEAN> in tiny_struct.
====
---- QUERY
# Ordering by struct column is not supported.
(impala) 03/04: IMPALA-12756: [DOCS] Unicode column name support documentation
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ab445195b0d4eb78eda16c7501f49e1fa554530e
Author: pranavyl <pr...@cloudera.com>
AuthorDate: Thu Dec 14 12:46:16 2023 -0800
IMPALA-12756: [DOCS] Unicode column name support documentation
The patch focuses on documenting that Impala supports unicode
column names, consistent with Hive's current support (as we use
Hive MetaStore to store table metadata).
Change-Id: I3d43d942a3ea069020f06adab6ea77e62ad5ffbe
Reviewed-on: http://gerrit.cloudera.org:8080/20950
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/topics/impala_identifiers.xml | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/docs/topics/impala_identifiers.xml b/docs/topics/impala_identifiers.xml
index 5f407fa69..b8760e374 100644
--- a/docs/topics/impala_identifiers.xml
+++ b/docs/topics/impala_identifiers.xml
@@ -51,20 +51,22 @@ under the License.
<li>
<p>
- The maximum length of an identifier is currently 128 characters, enforced by the metastore database.
+ The maximum length of an identifier is currently 128 characters except for column names which
+ can contain 767 characters, enforced by the metastore database.
</p>
</li>
<li>
<p>
- An identifier must start with an alphanumeric or underscore character. Quoting the identifier with
- backticks has no effect on the allowed characters in the name.
+ An identifier must start with an alphanumeric or underscore character except for column names which
+ can start with any unicode characters. Quoting the identifier with backticks has no effect on the allowed
+ characters in the name.
</p>
</li>
<li>
<p>
- An identifier can contain only ASCII characters.
+ An identifier can contain only ASCII characters except for column names which can contain unicode characters.
</p>
</li>