You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2023/03/23 04:14:43 UTC
[impala] branch master updated (fd4db07e7 -> f2cb2c9ce)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from fd4db07e7 Reapply IMPALA-11865: Set thread names for Java thread pools
new fe29a356b IMPALA-12011: Unset JAVA_TOOL_OPTIONS variable for webserver_private_key_password_cmd
new f2cb2c9ce IMPALA-11964: Make sure Impala returns error for Iceberg tables with equality deletes
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/util/webserver.cc | 5 +--
common/thrift/CatalogObjects.thrift | 3 +-
.../org/apache/impala/catalog/FeIcebergTable.java | 12 ++++--
.../impala/catalog/IcebergContentFileStore.java | 45 ++++++++++++++++-----
.../apache/impala/planner/IcebergScanPlanner.java | 15 +++++--
testdata/data/README | 6 +++
...78c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet | Bin
...80302-527b-4911-8c6e-88d416adac57-00001.parquet | Bin
.../0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro} | Bin 3675 -> 3933 bytes
.../8cbef400-daea-478a-858a-2baf2438f644-m0.avro | Bin 3619 -> 3617 bytes
...55-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro} | Bin 2189 -> 2314 bytes
...07-1-8cbef400-daea-478a-858a-2baf2438f644.avro} | Bin 2154 -> 2160 bytes
.../metadata/v1.metadata.json | 4 +-
.../metadata/v2.metadata.json | 8 ++--
.../metadata/version-hint.text | 0
.../functional/functional_schema_template.sql | 15 +++++++
.../datasets/functional/schema_constraints.csv | 1 +
.../queries/QueryTest/iceberg-negative.test | 21 ++++++++++
.../queries/QueryTest/iceberg-query.test | 10 +++++
19 files changed, 117 insertions(+), 28 deletions(-)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_delete_positional => iceberg_v2_delete_equality}/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet (100%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_delete_positional => iceberg_v2_delete_equality}/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet (100%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_positional_delete_all_rows/metadata/236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro => iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro} (84%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_delete_positional => iceberg_v2_delete_equality}/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro (91%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro => iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro} (82%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro => iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro} (88%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_delete_positional => iceberg_v2_delete_equality}/metadata/v1.metadata.json (92%)
copy testdata/data/iceberg_test/hadoop_catalog/ice/{iceberg_v2_delete_positional => iceberg_v2_delete_equality}/metadata/v2.metadata.json (88%)
copy testdata/data/iceberg_test/hadoop_catalog/{hadoop_catalog_test/functional_parquet/hadoop_catalog_test => ice/iceberg_v2_delete_equality}/metadata/version-hint.text (100%)
[impala] 02/02: IMPALA-11964: Make sure Impala returns error for Iceberg tables with equality deletes
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit f2cb2c9ceb833d331fe2f78d9d3c9a5bcaeff7dd
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Mar 7 18:43:53 2023 +0100
IMPALA-11964: Make sure Impala returns error for Iceberg tables with equality deletes
Impala only supports position deletes currently. It should raise an
error when equality deletes are encountered.
We already had a check for this when the query was planned by Iceberg.
But when we were using cached metadata the check was missing. This means
that Impala could return bogus results in the presence of equality
delete files. This patch adds check for the latter case as well.
Tables with equality delete files are still loadable by Impala, and
users can still query snapshots of it if they don't have equality
deletes.
Testing:
* added e2e tests
Change-Id: I14d7116692c0e47d0799be650dc323811e2ee0fb
Reviewed-on: http://gerrit.cloudera.org:8080/19601
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
common/thrift/CatalogObjects.thrift | 3 +-
.../org/apache/impala/catalog/FeIcebergTable.java | 12 ++-
.../impala/catalog/IcebergContentFileStore.java | 45 +++++++---
.../apache/impala/planner/IcebergScanPlanner.java | 15 +++-
testdata/data/README | 6 ++
...78c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet | Bin 0 -> 662 bytes
...80302-527b-4911-8c6e-88d416adac57-00001.parquet | Bin 0 -> 1581 bytes
.../0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro | Bin 0 -> 3933 bytes
.../8cbef400-daea-478a-858a-2baf2438f644-m0.avro | Bin 0 -> 3617 bytes
...755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro | Bin 0 -> 2314 bytes
...807-1-8cbef400-daea-478a-858a-2baf2438f644.avro | Bin 0 -> 2160 bytes
.../metadata/v1.metadata.json | 66 +++++++++++++++
.../metadata/v2.metadata.json | 93 +++++++++++++++++++++
.../metadata/version-hint.text | 1 +
.../functional/functional_schema_template.sql | 15 ++++
.../datasets/functional/schema_constraints.csv | 1 +
.../queries/QueryTest/iceberg-negative.test | 21 +++++
.../queries/QueryTest/iceberg-query.test | 10 +++
18 files changed, 269 insertions(+), 19 deletions(-)
diff --git a/common/thrift/CatalogObjects.thrift b/common/thrift/CatalogObjects.thrift
index 7dc214d3a..467191450 100644
--- a/common/thrift/CatalogObjects.thrift
+++ b/common/thrift/CatalogObjects.thrift
@@ -617,7 +617,8 @@ struct TIcebergPartitionStats {
struct TIcebergContentFileStore {
1: optional map<string, THdfsFileDesc> path_hash_to_data_file_without_deletes
2: optional map<string, THdfsFileDesc> path_hash_to_data_file_with_deletes
- 3: optional map<string, THdfsFileDesc> path_hash_to_delete_file
+ 3: optional map<string, THdfsFileDesc> path_hash_to_position_delete_file
+ 7: optional map<string, THdfsFileDesc> path_hash_to_equality_delete_file
4: optional bool has_avro
5: optional bool has_orc
6: optional bool has_parquet
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
index fbda3fd06..ca9e2e05c 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
@@ -712,11 +712,15 @@ public interface FeIcebergTable extends FeFsTable {
fileStore.addDataFileWithDeletes(pathHashAndFd.first, pathHashAndFd.second);
}
for (DeleteFile deleteFile : icebergFiles.deleteFiles) {
- Preconditions.checkState(
- deleteFile.content().equals(FileContent.EQUALITY_DELETES) ||
- deleteFile.content().equals(FileContent.POSITION_DELETES));
pathHashAndFd = getPathHashAndFd(deleteFile, table, hdfsFileDescMap);
- fileStore.addDeleteFileDescriptor(pathHashAndFd.first, pathHashAndFd.second);
+ if (deleteFile.content().equals(FileContent.POSITION_DELETES)) {
+ fileStore.addPositionDeleteFile(pathHashAndFd.first, pathHashAndFd.second);
+ } else if (deleteFile.content().equals(FileContent.EQUALITY_DELETES)) {
+ fileStore.addEqualityDeleteFile(pathHashAndFd.first, pathHashAndFd.second);
+ } else {
+ Preconditions.checkState(false,
+ "Delete file with unknown kind: " + deleteFile.path().toString());
+ }
}
return fileStore;
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java b/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
index d869665d5..8f169ae74 100644
--- a/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
+++ b/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
@@ -95,7 +95,8 @@ public class IcebergContentFileStore {
// Separate map-list containers for the different content files.
private MapListContainer dataFilesWithoutDeletes_ = new MapListContainer();
private MapListContainer dataFilesWithDeletes_ = new MapListContainer();
- private MapListContainer deleteFiles_ = new MapListContainer();
+ private MapListContainer positionDeleteFiles_ = new MapListContainer();
+ private MapListContainer equalityDeleteFiles_ = new MapListContainer();
// Caches file descriptors loaded during time-travel queries.
private final ConcurrentMap<String, FileDescriptor> oldFileDescMap_ =
@@ -120,8 +121,14 @@ public class IcebergContentFileStore {
}
}
- public void addDeleteFileDescriptor(String pathHash, FileDescriptor desc) {
- if (deleteFiles_.add(pathHash, desc)) {
+ public void addPositionDeleteFile(String pathHash, FileDescriptor desc) {
+ if (positionDeleteFiles_.add(pathHash, desc)) {
+ updateFileFormats(desc);
+ }
+ }
+
+ public void addEqualityDeleteFile(String pathHash, FileDescriptor desc) {
+ if (equalityDeleteFiles_.add(pathHash, desc)) {
updateFileFormats(desc);
}
}
@@ -139,7 +146,9 @@ public class IcebergContentFileStore {
}
public FileDescriptor getDeleteFileDescriptor(String pathHash) {
- return deleteFiles_.get(pathHash);
+ FileDescriptor ret = positionDeleteFiles_.get(pathHash);
+ if (ret != null) return ret;
+ return equalityDeleteFiles_.get(pathHash);
}
public FileDescriptor getOldFileDescriptor(String pathHash) {
@@ -154,19 +163,27 @@ public class IcebergContentFileStore {
return dataFilesWithDeletes_.getList();
}
- public List<FileDescriptor> getDeleteFiles() { return deleteFiles_.getList(); }
+ public List<FileDescriptor> getPositionDeleteFiles() {
+ return positionDeleteFiles_.getList();
+ }
+
+ public List<FileDescriptor> getEqualityDeleteFiles() {
+ return equalityDeleteFiles_.getList();
+ }
public long getNumFiles() {
return dataFilesWithoutDeletes_.getNumFiles() +
dataFilesWithDeletes_.getNumFiles() +
- deleteFiles_.getNumFiles();
+ positionDeleteFiles_.getNumFiles() +
+ equalityDeleteFiles_.getNumFiles();
}
public Iterable<FileDescriptor> getAllFiles() {
return Iterables.concat(
dataFilesWithoutDeletes_.getList(),
dataFilesWithDeletes_.getList(),
- deleteFiles_.getList());
+ positionDeleteFiles_.getList(),
+ equalityDeleteFiles_.getList());
}
public Iterable<FileDescriptor> getAllDataFiles() {
@@ -194,7 +211,8 @@ public class IcebergContentFileStore {
TIcebergContentFileStore ret = new TIcebergContentFileStore();
ret.setPath_hash_to_data_file_without_deletes(dataFilesWithoutDeletes_.toThrift());
ret.setPath_hash_to_data_file_with_deletes(dataFilesWithDeletes_.toThrift());
- ret.setPath_hash_to_delete_file(deleteFiles_.toThrift());
+ ret.setPath_hash_to_position_delete_file(positionDeleteFiles_.toThrift());
+ ret.setPath_hash_to_equality_delete_file(equalityDeleteFiles_.toThrift());
ret.setHas_avro(hasAvro_);
ret.setHas_orc(hasOrc_);
ret.setHas_parquet(hasParquet_);
@@ -215,9 +233,14 @@ public class IcebergContentFileStore {
tFileStore.getPath_hash_to_data_file_with_deletes(),
networkAddresses, hostIndex);
}
- if (tFileStore.isSetPath_hash_to_delete_file()) {
- ret.deleteFiles_ = MapListContainer.fromThrift(
- tFileStore.getPath_hash_to_delete_file(),
+ if (tFileStore.isSetPath_hash_to_position_delete_file()) {
+ ret.positionDeleteFiles_ = MapListContainer.fromThrift(
+ tFileStore.getPath_hash_to_position_delete_file(),
+ networkAddresses, hostIndex);
+ }
+ if (tFileStore.isSetPath_hash_to_equality_delete_file()) {
+ ret.equalityDeleteFiles_ = MapListContainer.fromThrift(
+ tFileStore.getPath_hash_to_equality_delete_file(),
networkAddresses, hostIndex);
}
ret.hasAvro_ = tFileStore.isSetHas_avro() ? tFileStore.isHas_avro() : false;
diff --git a/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
index dea0ebcc2..29accb498 100644
--- a/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
@@ -158,11 +158,20 @@ public class IcebergScanPlanner {
tblRef_.getTimeTravelSpec() != null;
}
- private void setFileDescriptorsBasedOnFileStore() {
+ private void setFileDescriptorsBasedOnFileStore() throws ImpalaException {
IcebergContentFileStore fileStore = getIceTable().getContentFileStore();
+ if (!fileStore.getEqualityDeleteFiles().isEmpty()) {
+ // TODO(IMPALA-11388): Add support for equality deletes.
+ FileDescriptor firstEqualityDeleteFile = fileStore.getEqualityDeleteFiles().get(0);
+ throw new ImpalaRuntimeException(String.format(
+ "Iceberg table %s has EQUALITY delete file which is currently " +
+ "not supported by Impala, for example: %s",
+ getIceTable().getFullName(),
+ firstEqualityDeleteFile.getAbsolutePath(getIceTable().getLocation())));
+ }
dataFilesWithoutDeletes_ = fileStore.getDataFilesWithoutDeletes();
dataFilesWithDeletes_ = fileStore.getDataFilesWithDeletes();
- deleteFiles_ = new HashSet<>(fileStore.getDeleteFiles());
+ deleteFiles_ = new HashSet<>(fileStore.getPositionDeleteFiles());
updateDeleteStatistics();
}
@@ -342,7 +351,7 @@ public class IcebergScanPlanner {
if (delFile.content() == FileContent.EQUALITY_DELETES) {
throw new ImpalaRuntimeException(String.format(
"Iceberg table %s has EQUALITY delete file which is currently " +
- "not supported by Impala: %s", getIceTable().getFullName(),
+ "not supported by Impala, for example: %s", getIceTable().getFullName(),
delFile.path()));
}
Pair<FileDescriptor, Boolean> delFileDesc = getFileDescriptor(delFile);
diff --git a/testdata/data/README b/testdata/data/README
index b52ba0c4e..ac22c7cf0 100644
--- a/testdata/data/README
+++ b/testdata/data/README
@@ -747,6 +747,12 @@ to make these tables correspond to an Iceberg table in a HadoopCatalog instead o
HiveCatalog.
The table has a positional delete file.
+iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality:
+Since Hive/Spark is unable to write equality delete files we've copied the contents of
+'iceberg_v2_delete_positional' and manually edited the metadata to have equality delete
+files in it. Only modified the metadata files, the actual delete files are still
+positional.
+
iceberg_test/iceberg_migrated_alter_test
Generated and migrated by Hive
CREATE TABLE iceberg_migrated_alter_test (int_col int, string_col string, double_col double) stored as parquet;
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet
new file mode 100644
index 000000000..d27d6b3f6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet
new file mode 100644
index 000000000..2fb34f0b4
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro
new file mode 100644
index 000000000..5fa4ae11d
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro
new file mode 100644
index 000000000..a15eff13b
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro
new file mode 100644
index 000000000..7c849b537
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro
new file mode 100644
index 000000000..c90a29cc0
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json
new file mode 100644
index 000000000..b658df4e0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json
@@ -0,0 +1,66 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "3deb545a-5a19-48f1-ad07-a4d80c677e3e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1649071501670,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "long"
+ }, {
+ "id" : 2,
+ "name" : "data",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "owner" : "tamasmate",
+ "write.delete.mode" : "merge-on-read"
+ },
+ "current-snapshot-id" : 6816997371555012807,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 6816997371555012807,
+ "timestamp-ms" : 1649071501670,
+ "summary" : {
+ "operation" : "append",
+ "spark.app.id" : "local-1649071493099",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "662",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "662",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1649071501670,
+ "snapshot-id" : 6816997371555012807
+ } ],
+ "metadata-log" : [ ]
+}
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json
new file mode 100644
index 000000000..bed83e8b8
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json
@@ -0,0 +1,93 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "3deb545a-5a19-48f1-ad07-a4d80c677e3e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1649071557501,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "long"
+ }, {
+ "id" : 2,
+ "name" : "data",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "owner" : "tamasmate",
+ "write.delete.mode" : "merge-on-read"
+ },
+ "current-snapshot-id" : 5725822353600261755,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 6816997371555012807,
+ "timestamp-ms" : 1649071501670,
+ "summary" : {
+ "operation" : "append",
+ "spark.app.id" : "local-1649071493099",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "662",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "662",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5725822353600261755,
+ "parent-snapshot-id" : 6816997371555012807,
+ "timestamp-ms" : 1649071557501,
+ "summary" : {
+ "operation" : "overwrite",
+ "spark.app.id" : "local-1649071493099",
+ "added-delete-files" : "1",
+ "added-files-size" : "1598",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "2260",
+ "total-data-files" : "1",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1649071501670,
+ "snapshot-id" : 6816997371555012807
+ }, {
+ "timestamp-ms" : 1649071557501,
+ "snapshot-id" : 5725822353600261755
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1649071501670,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json"
+ } ]
+}
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text
new file mode 100644
index 000000000..d8263ee98
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text
@@ -0,0 +1 @@
+2
\ No newline at end of file
diff --git a/testdata/datasets/functional/functional_schema_template.sql b/testdata/datasets/functional/functional_schema_template.sql
index 68d5453dd..ada17381a 100644
--- a/testdata/datasets/functional/functional_schema_template.sql
+++ b/testdata/datasets/functional/functional_schema_template.sql
@@ -3466,6 +3466,21 @@ hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/i
---- DATASET
functional
---- BASE_TABLE_NAME
+iceberg_v2_delete_equality
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('write.format.default'='parquet', 'iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_delete_equality',
+ 'format-version'='2', 'write.update.mode'='merge-on-read');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
iceberg_multiple_storage_locations
---- CREATE
CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
diff --git a/testdata/datasets/functional/schema_constraints.csv b/testdata/datasets/functional/schema_constraints.csv
index 6cf3b16a0..4c23e23dd 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -82,6 +82,7 @@ table_name:iceberg_timestamp_part, constraint:restrict_to, table_format:parquet/
table_name:iceberg_timestamptz_part, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_uppercase_col, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_v2_delete_positional, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_delete_equality, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_v2_no_deletes, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_v2_no_deletes_orc, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_v2_positional_update_all_rows, constraint:restrict_to, table_format:parquet/none/none
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
index 2231c8e15..a4c1836f4 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
@@ -662,3 +662,24 @@ select * from functional_parquet.iceberg_alltypes_part for system_time as of '20
---- CATCH
IllegalArgumentException: Cannot find a snapshot older than 2000-01-01 01:02:03
====
+---- QUERY
+# Querying a table with equality deletes is not allowed.
+# We don't use time-travel, so we plan the query from cached metadata.
+select * from functional_parquet.iceberg_v2_delete_equality;
+---- CATCH
+ImpalaRuntimeException: Iceberg table functional_parquet.iceberg_v2_delete_equality has EQUALITY delete file which is currently not supported by Impala
+====
+---- QUERY
+# Querying a table with equality deletes is not allowed.
+# Use time-travel based on snapshot id.
+select * from functional_parquet.iceberg_v2_delete_equality for system_version as of 5725822353600261755;
+---- CATCH
+ImpalaRuntimeException: Iceberg table functional_parquet.iceberg_v2_delete_equality has EQUALITY delete file which is currently not supported by Impala
+====
+---- QUERY
+# Querying a table with equality deletes is not allowed.
+# Use time-travel based on timestamp.
+select * from functional_parquet.iceberg_v2_delete_equality for system_time as of now();
+---- CATCH
+ImpalaRuntimeException: Iceberg table functional_parquet.iceberg_v2_delete_equality has EQUALITY delete file which is currently not supported by Impala
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
index a6b28cfc8..b90f02c0b 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
@@ -1160,3 +1160,13 @@ where i.action in ('view') and j.id=1 and j.id=i.id;
---- TYPES
int
====
+---- QUERY
+# We can query a snapshot if it doesn't have equality deletes.
+select * from iceberg_v2_delete_equality for system_version as of 6816997371555012807;
+---- RESULTS
+1,'a'
+2,'b'
+3,'c'
+---- TYPES
+BIGINT,STRING
+====
[impala] 01/02: IMPALA-12011: Unset JAVA_TOOL_OPTIONS variable for webserver_private_key_password_cmd
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit fe29a356b91b3a13b95a3ecbaf2d161b556e93ef
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Tue Mar 21 15:23:55 2023 +0800
IMPALA-12011: Unset JAVA_TOOL_OPTIONS variable for webserver_private_key_password_cmd
IMPALA-6293 intends to unset JAVA_TOOL_OPTIONS variable for all the
shell commands that run java. So they won't fail in binding to the same
debug port. However, it's not done correctly for the command of
webserver_private_key_password_cmd. This patch corrects the mistake.
Tests:
- Tested on a secured cluster. Verified catalogd can start with port
binding options set in JAVA_TOOL_OPTIONS.
Change-Id: I40b57a8eb28e6e2b48b35b0f361f0e220c7f89d2
Reviewed-on: http://gerrit.cloudera.org:8080/19638
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/util/webserver.cc | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/be/src/util/webserver.cc b/be/src/util/webserver.cc
index 5051d17f5..9d2716591 100644
--- a/be/src/util/webserver.cc
+++ b/be/src/util/webserver.cc
@@ -419,9 +419,8 @@ Status Webserver::Start() {
const string& password_cmd = FLAGS_webserver_private_key_password_cmd;
if (!password_cmd.empty()) {
- if (!RunShellProcess(password_cmd, &key_password, true)) {
- return Status(TErrorCode::SSL_PASSWORD_CMD_FAILED, password_cmd, key_password,
- {"JAVA_TOOL_OPTIONS"});
+ if (!RunShellProcess(password_cmd, &key_password, true, {"JAVA_TOOL_OPTIONS"})) {
+ return Status(TErrorCode::SSL_PASSWORD_CMD_FAILED, password_cmd, key_password);
}
options.push_back("ssl_private_key_password");
options.push_back(key_password.c_str());