You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/01/18 00:48:22 UTC
[impala] branch master updated (79f4c05 -> 3f51a6a)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 79f4c05 IMPALA-10951 (part 2): Upgrade protobuf library to 3.14.0 for Impala
new 577fc2e IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling
new 3f51a6a IMPALA-11051: Add support for 'void' Iceberg partition transform
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/hdfs-table-sink.cc | 2 +-
be/src/runtime/descriptors.cc | 5 +-
be/src/runtime/descriptors.h | 6 +-
common/thrift/CatalogObjects.thrift | 1 +
.../org/apache/impala/analysis/InsertStmt.java | 12 ++-
.../java/org/apache/impala/util/IcebergUtil.java | 24 ++++-
.../QueryTest/iceberg-partitioned-insert.test | 104 +++++++++++++++++++++
.../queries/QueryTest/show-create-table.test | 12 +++
.../queries/QueryTest/spilling.test | 2 +-
9 files changed, 156 insertions(+), 12 deletions(-)
[impala] 01/02: IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 577fc2ee21a8cb6fc49f9df8367d63ebccc13e61
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Mon Jan 10 10:08:41 2022 -0800
IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling
The first test case in TestSpillingDebugActionDimensions.test_spilling
has been flaky for not spilling any partitions in its hash join node.
This patch fixes the flakiness by reducing the buffer_pool_limit from
215 MB to 110 MB, which is around double of the query Per Host Min
Memory Reservation.
Testing:
- Manually run the first test case of
TestSpillingDebugActionDimensions.test_spilling. Verify that both of
the hash joins are spilling and the test pass.
Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077
Reviewed-on: http://gerrit.cloudera.org:8080/18138
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/workloads/functional-query/queries/QueryTest/spilling.test | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/spilling.test b/testdata/workloads/functional-query/queries/QueryTest/spilling.test
index 8a4a36b..2632953 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/spilling.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/spilling.test
@@ -1,6 +1,6 @@
====
---- QUERY
-set buffer_pool_limit=215m;
+set buffer_pool_limit=110m;
select count(l1.l_tax)
from
lineitem l1,
[impala] 02/02: IMPALA-11051: Add support for 'void' Iceberg partition transform
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 3f51a6a761bcb93675088afc155ccc3d2a380401
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Wed Dec 15 11:51:49 2021 +0100
IMPALA-11051: Add support for 'void' Iceberg partition transform
Iceberg recently added a new partition transform called 'void':
https://iceberg.apache.org/#spec/#partition-transforms
This patch adds support for this transform.
When the user wants to drop a column from the partition spec,
the VOID transform should be used instead of just omitting
the column. Simply omitting the column might cause problems when
the metadata table is being queried (currently only supported
by other engines).
Testing
* added SHOW CREATE TABLE test
* added e2e test
Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0
Reviewed-on: http://gerrit.cloudera.org:8080/18102
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/exec/hdfs-table-sink.cc | 2 +-
be/src/runtime/descriptors.cc | 5 +-
be/src/runtime/descriptors.h | 6 +-
common/thrift/CatalogObjects.thrift | 1 +
.../org/apache/impala/analysis/InsertStmt.java | 12 ++-
.../java/org/apache/impala/util/IcebergUtil.java | 24 ++++-
.../QueryTest/iceberg-partitioned-insert.test | 104 +++++++++++++++++++++
.../queries/QueryTest/show-create-table.test | 12 +++
8 files changed, 155 insertions(+), 11 deletions(-)
diff --git a/be/src/exec/hdfs-table-sink.cc b/be/src/exec/hdfs-table-sink.cc
index 8a72d6c..75b8a5b 100644
--- a/be/src/exec/hdfs-table-sink.cc
+++ b/be/src/exec/hdfs-table-sink.cc
@@ -500,7 +500,7 @@ Status HdfsTableSink::CreateNewTmpFile(RuntimeState* state,
string HdfsTableSink::GetPartitionName(int i) {
if (IsIceberg()) {
DCHECK_LT(i, partition_key_expr_evals_.size());
- return table_desc_->IcebergPartitionNames()[i];
+ return table_desc_->IcebergNonVoidPartitionNames()[i];
} else {
DCHECK_LT(i, table_desc_->num_clustering_cols());
return table_desc_->col_descs()[i].name();
diff --git a/be/src/runtime/descriptors.cc b/be/src/runtime/descriptors.cc
index eb5bd1d..9fa7074 100644
--- a/be/src/runtime/descriptors.cc
+++ b/be/src/runtime/descriptors.cc
@@ -254,7 +254,10 @@ HdfsTableDescriptor::HdfsTableDescriptor(const TTableDescriptor& tdesc, ObjectPo
const TIcebergPartitionSpec& spec = tdesc.icebergTable.partition_spec[
tdesc.icebergTable.default_partition_spec_id];
for (const TIcebergPartitionField& spec_field : spec.partition_fields) {
- iceberg_partition_names_.push_back(spec_field.field_name);
+ if (spec_field.transform.transform_type == TIcebergPartitionTransformType::VOID) {
+ continue;
+ }
+ iceberg_non_void_partition_names_.push_back(spec_field.field_name);
}
iceberg_parquet_compression_codec_ = tdesc.icebergTable.parquet_compression_codec;
iceberg_parquet_row_group_size_ = tdesc.icebergTable.parquet_row_group_size;
diff --git a/be/src/runtime/descriptors.h b/be/src/runtime/descriptors.h
index 3c9d9e7..e4d5326 100644
--- a/be/src/runtime/descriptors.h
+++ b/be/src/runtime/descriptors.h
@@ -342,8 +342,8 @@ class HdfsTableDescriptor : public TableDescriptor {
bool IsIcebergTable() const { return is_iceberg_; }
const std::string& IcebergTableLocation() const { return iceberg_table_location_; }
- const std::vector<std::string>& IcebergPartitionNames() const {
- return iceberg_partition_names_;
+ const std::vector<std::string>& IcebergNonVoidPartitionNames() const {
+ return iceberg_non_void_partition_names_;
}
const TCompressionCodec& IcebergParquetCompressionCodec() const {
return iceberg_parquet_compression_codec_;
@@ -374,7 +374,7 @@ class HdfsTableDescriptor : public TableDescriptor {
TValidWriteIdList valid_write_id_list_;
bool is_iceberg_ = false;
std::string iceberg_table_location_;
- std::vector<std::string> iceberg_partition_names_;
+ std::vector<std::string> iceberg_non_void_partition_names_;
TCompressionCodec iceberg_parquet_compression_codec_;
int64_t iceberg_parquet_row_group_size_;
int64_t iceberg_parquet_plain_page_size_;
diff --git a/common/thrift/CatalogObjects.thrift b/common/thrift/CatalogObjects.thrift
index 3d94f6e..1cc729e 100644
--- a/common/thrift/CatalogObjects.thrift
+++ b/common/thrift/CatalogObjects.thrift
@@ -140,6 +140,7 @@ enum TIcebergPartitionTransformType {
YEAR = 4
BUCKET = 5
TRUNCATE = 6
+ VOID = 7
}
struct TCompressionCodec {
diff --git a/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java b/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
index 09adeb9..cec954b 100644
--- a/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
@@ -881,9 +881,14 @@ public class InsertStmt extends StatementBase {
}
}
- if (isIcebergTarget()) {
- Preconditions.checkState(
- partitionKeyExprs_.size() == icebergPartSpec.getIcebergPartitionFieldsSize());
+ if (isIcebergTarget() && icebergPartSpec.hasPartitionFields()) {
+ int parts = 0;
+ for (IcebergPartitionField pField : icebergPartSpec.getIcebergPartitionFields()) {
+ if (pField.getTransformType() != TIcebergPartitionTransformType.VOID) {
+ ++parts;
+ }
+ }
+ Preconditions.checkState(partitionKeyExprs_.size() == parts);
}
else if (isKuduTable) {
Preconditions.checkState(
@@ -964,6 +969,7 @@ public class InsertStmt extends StatementBase {
IcebergPartitionSpec icebergPartSpec) throws AnalysisException {
if (!icebergPartSpec.hasPartitionFields()) return;
for (IcebergPartitionField partField : icebergPartSpec.getIcebergPartitionFields()) {
+ if (partField.getTransformType() == TIcebergPartitionTransformType.VOID) continue;
for (int i = 0; i < selectListExprs.size(); ++i) {
IcebergColumn targetColumn = (IcebergColumn)selectExprTargetColumns.get(i);
if (targetColumn.getFieldId() != partField.getSourceId()) continue;
diff --git a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
index 364430f..0b001d2 100644
--- a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
@@ -223,6 +223,8 @@ public class IcebergUtil {
} else if (transformType == TIcebergPartitionTransformType.TRUNCATE) {
builder.truncate(partitionField.getField_name(),
partitionField.getTransform().getTransform_param());
+ } else if (transformType == TIcebergPartitionTransformType.VOID) {
+ builder.alwaysNull(partitionField.getField_name());
} else {
throw new ImpalaRuntimeException(String.format("Skip partition: %s, %s",
partitionField.getField_name(), transformType));
@@ -409,6 +411,7 @@ public class IcebergUtil {
case "DAY": case "DAYS": return TIcebergPartitionTransformType.DAY;
case "MONTH": case "MONTHS": return TIcebergPartitionTransformType.MONTH;
case "YEAR": case "YEARS": return TIcebergPartitionTransformType.YEAR;
+ case "VOID": return TIcebergPartitionTransformType.VOID;
default:
throw new TableLoadingException("Unsupported iceberg partition type: " +
transformType);
@@ -485,6 +488,14 @@ public class IcebergUtil {
TIcebergPartitionTransformType.HOUR);
return new Pair<String, Integer>(mappingKey, null);
}
+
+ @Override
+ public Pair<String, Integer> alwaysNull(int fieldId, String sourceName,
+ int sourceId) {
+ String mappingKey = getPartitonTransformMappingKey(sourceId,
+ TIcebergPartitionTransformType.VOID);
+ return new Pair<String, Integer>(mappingKey, null);
+ }
});
// Move the content of the List into a HashMap for faster querying in the future.
HashMap<String, Integer> result = Maps.newHashMap();
@@ -643,14 +654,20 @@ public class IcebergUtil {
PartitionData data = new PartitionData(spec.getIcebergPartitionFieldsSize());
String[] partitions = path.split("/", -1);
- for (int i = 0; i < partitions.length; i += 1) {
+ int path_i = 0;
+ for (int i = 0; i < spec.getIcebergPartitionFieldsSize(); ++i) {
IcebergPartitionField field = spec.getIcebergPartitionFields().get(i);
- String[] parts = partitions[i].split("=", 2);
+ if (field.getTransformType() == TIcebergPartitionTransformType.VOID) {
+ continue;
+ }
+ String[] parts = partitions[path_i].split("=", 2);
Preconditions.checkArgument(parts.length == 2 && parts[0] != null &&
- field.getFieldName().equals(parts[0]), "Invalid partition: %s", partitions[i]);
+ field.getFieldName().equals(parts[0]), "Invalid partition: %s",
+ partitions[path_i]);
TIcebergPartitionTransformType transformType = field.getTransformType();
data.set(i, getPartitionValue(
partitionType.fields().get(i).type(), transformType, parts[1]));
+ path_i += 1;
}
return data;
}
@@ -676,6 +693,7 @@ public class IcebergUtil {
case YEAR: return parseYearToTransformYear(stringValue);
case MONTH: return parseMonthToTransformMonth(stringValue);
case HOUR: return parseHourToTransformHour(stringValue);
+ case VOID: return null;
}
throw new ImpalaRuntimeException("Unexpected partition transform: " + transformType);
}
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
index c8d3a68..090edff 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test
@@ -315,3 +315,107 @@ row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_alter_part/data/d_year=2020
---- TYPES
STRING, STRING, STRING
====
+---- QUERY
+create table ice_void (i int, s string, d date)
+partitioned by spec (void(i), truncate(1, s), year(d))
+stored as iceberg;
+insert into ice_void values (1, 'one', '2001-01-01'),
+ (11,'other','2001-01-11'),
+ (2, 'two', '2002-02-02'),
+ (3, 'three', '2003-03-03');
+select * from ice_void;
+---- RESULTS
+1,'one',2001-01-01
+11,'other',2001-01-11
+2,'two',2002-02-02
+3,'three',2003-03-03
+---- TYPES
+INT, STRING, DATE
+====
+---- QUERY
+show files in ice_void;
+---- RESULTS: VERIFY_IS_SUBSET
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=o/d_year=2001/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2002/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2003/.*.0.parq','.*',''
+---- TYPES
+STRING, STRING, STRING
+====
+---- QUERY
+alter table ice_void set partition spec (i, void(s), year(d));
+insert into ice_void values (4, 'four', '2004-04-04');
+select * from ice_void;
+---- RESULTS
+1,'one',2001-01-01
+11,'other',2001-01-11
+2,'two',2002-02-02
+3,'three',2003-03-03
+4,'four',2004-04-04
+---- TYPES
+INT, STRING, DATE
+====
+---- QUERY
+show files in ice_void;
+---- RESULTS: VERIFY_IS_SUBSET
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=o/d_year=2001/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2002/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2003/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/i=4/d_year=2004/.*.0.parq','.*',''
+---- TYPES
+STRING, STRING, STRING
+====
+---- QUERY
+alter table ice_void set partition spec (i, void(s), void(d));
+insert into ice_void values (5, 'five', '2005-05-05'), (5, 'other five', '1995-05-05');
+select * from ice_void;
+---- RESULTS
+1,'one',2001-01-01
+11,'other',2001-01-11
+2,'two',2002-02-02
+3,'three',2003-03-03
+4,'four',2004-04-04
+5,'five',2005-05-05
+5,'other five',1995-05-05
+---- TYPES
+INT, STRING, DATE
+====
+---- QUERY
+show files in ice_void;
+---- RESULTS: VERIFY_IS_SUBSET
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=o/d_year=2001/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2002/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2003/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/i=4/d_year=2004/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/i=5/[^=/]*.0.parq','.*',''
+---- TYPES
+STRING, STRING, STRING
+====
+---- QUERY
+alter table ice_void set partition spec (void(i), void(s), void(d));
+insert into ice_void values (6, 'six', '2006-06-06'), (7, 'seven', '2007-07-07');
+select * from ice_void;
+---- RESULTS
+1,'one',2001-01-01
+11,'other',2001-01-11
+2,'two',2002-02-02
+3,'three',2003-03-03
+4,'four',2004-04-04
+5,'five',2005-05-05
+5,'other five',1995-05-05
+6,'six',2006-06-06
+7,'seven',2007-07-07
+---- TYPES
+INT, STRING, DATE
+====
+---- QUERY
+show files in ice_void;
+---- RESULTS: VERIFY_IS_SUBSET
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=o/d_year=2001/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2002/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/s_trunc=t/d_year=2003/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/i=4/d_year=2004/.*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/i=5/[^=/]*.0.parq','.*',''
+row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_void/data/[^=/]*.0.parq','.*',''
+---- TYPES
+STRING, STRING, STRING
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/show-create-table.test b/testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
index dfe435e..81c67d3 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
@@ -926,3 +926,15 @@ LOCATION '$$location_uri$$'
TBLPROPERTIES ('external.table.purge'='TRUE', 'write.format.default'='parquet',
'engine.hive.enabled'='true', 'iceberg.catalog'='ice_hadoop_cat')
====
+---- CREATE_TABLE
+CREATE TABLE iceberg_void_transform (i int, j int)
+PARTITIONED BY SPEC (void(i), void(j))
+STORED AS ICEBERG
+---- RESULTS-HIVE-3
+CREATE EXTERNAL TABLE show_create_table_test_db.iceberg_void_transform (i INT NULL, j INT NULL)
+PARTITIONED BY SPEC (VOID(i), VOID(j))
+STORED AS ICEBERG
+LOCATION '$$location_uri$$'
+TBLPROPERTIES ('external.table.purge'='TRUE', 'write.format.default'='parquet',
+'engine.hive.enabled'='true', 'table_type'='ICEBERG')
+====