You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2023/09/06 16:15:04 UTC
[impala] branch master updated: IMPALA-12407: Add test table with Iceberg Equality deletes
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 707f149c9 IMPALA-12407: Add test table with Iceberg Equality deletes
707f149c9 is described below
commit 707f149c9449d15f54dd7229f79a21c0793f2af5
Author: Tamas Mate <tm...@apache.org>
AuthorDate: Fri Aug 25 16:06:02 2023 +0200
IMPALA-12407: Add test table with Iceberg Equality deletes
This commit updates the iceberg_v2_delete_equality test table. The
previous table was a modified positional delete table and the delete
files were actually positional delete files.
This table was created with Flink, which is one of the few services that
can write equality delete files currently.
Testing:
- Queried the table manually from Hive/Impala
- Ran the related E2E tests
Change-Id: I2d7e5928aff95ed09b1d7725b31a1698e3c31835
Reviewed-on: http://gerrit.cloudera.org:8080/20422
Reviewed-by: Gabor Kaszab <ga...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/data/README | 25 +++-
...ec696-613a-4270-a1f1-1f69947622d4-00001.parquet | Bin 0 -> 710 bytes
...ec696-613a-4270-a1f1-1f69947622d4-00002.parquet | Bin 0 -> 458 bytes
...d6d35-d95e-468f-a7d9-d64f7a7b5282-00001.parquet | Bin 0 -> 709 bytes
...d6d35-d95e-468f-a7d9-d64f7a7b5282-00002.parquet | Bin 0 -> 457 bytes
...c4bad-51b8-4c65-a20e-4bff5b1726b7-00001.parquet | Bin 0 -> 729 bytes
...c4bad-51b8-4c65-a20e-4bff5b1726b7-00002.parquet | Bin 0 -> 457 bytes
...78c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet | Bin 662 -> 0 bytes
...80302-527b-4911-8c6e-88d416adac57-00001.parquet | Bin 1581 -> 0 bytes
.../0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro | Bin 3933 -> 0 bytes
.../22029492-dcdb-4ba1-a447-9b9f7200f9e4-m0.avro | Bin 0 -> 3720 bytes
.../22029492-dcdb-4ba1-a447-9b9f7200f9e4-m1.avro | Bin 0 -> 3688 bytes
.../30efd05a-a329-4420-8592-9fab221add9e-m0.avro | Bin 0 -> 3720 bytes
.../30efd05a-a329-4420-8592-9fab221add9e-m1.avro | Bin 0 -> 3688 bytes
.../490632c7-54bf-4362-97e7-b75ab18da9ac-m0.avro | Bin 0 -> 3727 bytes
.../490632c7-54bf-4362-97e7-b75ab18da9ac-m1.avro | Bin 0 -> 3689 bytes
.../8cbef400-daea-478a-858a-2baf2438f644-m0.avro | Bin 3617 -> 0 bytes
...54-2-30efd05a-a329-4420-8592-9fab221add9e.avro} | Bin 2314 -> 2618 bytes
...35-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro} | Bin 2160 -> 2312 bytes
...91-3-490632c7-54bf-4362-97e7-b75ab18da9ac.avro} | Bin 2314 -> 2926 bytes
.../metadata/v1.metadata.json | 56 +++-----
.../metadata/v2.metadata.json | 90 ++++++------
.../metadata/v3.metadata.json | 120 ++++++++++++++++
.../metadata/v4.metadata.json | 151 +++++++++++++++++++++
.../metadata/version-hint.text | 1 -
.../metadata/version-hint.txt | 1 +
.../queries/QueryTest/iceberg-negative.test | 2 +-
.../queries/QueryTest/iceberg-query.test | 7 +-
28 files changed, 362 insertions(+), 91 deletions(-)
diff --git a/testdata/data/README b/testdata/data/README
index c5f8f4ed0..c885d81bd 100644
--- a/testdata/data/README
+++ b/testdata/data/README
@@ -748,10 +748,27 @@ HiveCatalog.
The table has a positional delete file.
iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality:
-Since Hive/Spark is unable to write equality delete files we've copied the contents of
-'iceberg_v2_delete_positional' and manually edited the metadata to have equality delete
-files in it. Only modified the metadata files, the actual delete files are still
-positional.
+Since Hive/Spark is unable to write equality delete files Flink has been used to create
+an equality delete file by overwriting an existing unique row with the following
+statements:
+ CREATE TABLE `ssb`.`ssb_default`.`iceberg_v2_delete_equality` (
+ `id` BIGINT UNIQUE COMMENT 'unique id',
+ `data` STRING NOT NULL,
+ PRIMARY KEY(`id`) NOT ENFORCED
+ ) with ('format-version'='2',
+ 'write.upsert.enabled'='true',
+ 'connector' = 'iceberg',
+ 'catalog-database' = 'test_db',
+ 'catalog-type' = 'hive',
+ 'catalog-name' = 'iceberg_hive_catalog',
+ 'hive-conf-dir' = '/etc/hive/conf',
+ 'engine.hive.enabled' = 'true'
+ );
+ insert into iceberg_v2_delete_equality values (1, 'test_1_base');
+ insert into iceberg_v2_delete_equality values (2, 'test_2_base');
+ insert into iceberg_v2_delete_equality values (2, 'test_2_updated');
+This table was created with HDFS absolute paths, which were replaced with the script
+specified in `iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes`.
iceberg_test/iceberg_migrated_alter_test
Generated and migrated by Hive
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00001.parquet
new file mode 100644
index 000000000..f686c568d
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00002.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00002.parquet
new file mode 100644
index 000000000..c55b8b2c9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-231ec696-613a-4270-a1f1-1f69947622d4-00002.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00001.parquet
new file mode 100644
index 000000000..86bd2792d
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00002.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00002.parquet
new file mode 100644
index 000000000..acec892a5
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-931d6d35-d95e-468f-a7d9-d64f7a7b5282-00002.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00001.parquet
new file mode 100644
index 000000000..6713d9b54
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00002.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00002.parquet
new file mode 100644
index 000000000..acec892a5
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-e3ac4bad-51b8-4c65-a20e-4bff5b1726b7-00002.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet
deleted file mode 100644
index d27d6b3f6..000000000
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00000-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-00001.parquet and /dev/null differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet
deleted file mode 100644
index 2fb34f0b4..000000000
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet and /dev/null differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro
deleted file mode 100644
index 5fa4ae11d..000000000
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro and /dev/null differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m0.avro
new file mode 100644
index 000000000..37e52bf37
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m1.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m1.avro
new file mode 100644
index 000000000..b76958680
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/22029492-dcdb-4ba1-a447-9b9f7200f9e4-m1.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m0.avro
new file mode 100644
index 000000000..c717a646e
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m1.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m1.avro
new file mode 100644
index 000000000..eac09839f
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/30efd05a-a329-4420-8592-9fab221add9e-m1.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m0.avro
new file mode 100644
index 000000000..23f8295e0
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m1.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m1.avro
new file mode 100644
index 000000000..9bd9f69fe
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/490632c7-54bf-4362-97e7-b75ab18da9ac-m1.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro
deleted file mode 100644
index a15eff13b..000000000
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro and /dev/null differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-3207673167795000154-2-30efd05a-a329-4420-8592-9fab221add9e.avro
similarity index 73%
copy from testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro
copy to testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-3207673167795000154-2-30efd05a-a329-4420-8592-9fab221add9e.avro
index 7c849b537..de1e9e436 100644
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-3207673167795000154-2-30efd05a-a329-4420-8592-9fab221add9e.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro
similarity index 83%
rename from testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro
rename to testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro
index c90a29cc0..6f5cfe142 100644
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5763349507283783091-3-490632c7-54bf-4362-97e7-b75ab18da9ac.avro
similarity index 65%
rename from testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro
rename to testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5763349507283783091-3-490632c7-54bf-4362-97e7-b75ab18da9ac.avro
index 7c849b537..7e52eaa9b 100644
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5763349507283783091-3-490632c7-54bf-4362-97e7-b75ab18da9ac.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json
index b658df4e0..a02e4d730 100644
--- a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json
@@ -1,23 +1,24 @@
{
"format-version" : 2,
- "table-uuid" : "3deb545a-5a19-48f1-ad07-a4d80c677e3e",
+ "table-uuid" : "580ae1f6-35ec-444a-9a34-623227d05f22",
"location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
- "last-sequence-number" : 1,
- "last-updated-ms" : 1649071501670,
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1692789322627,
"last-column-id" : 2,
"current-schema-id" : 0,
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
+ "identifier-field-ids" : [ 1 ],
"fields" : [ {
"id" : 1,
"name" : "id",
- "required" : false,
+ "required" : true,
"type" : "long"
}, {
"id" : 2,
"name" : "data",
- "required" : false,
+ "required" : true,
"type" : "string"
} ]
} ],
@@ -33,34 +34,21 @@
"fields" : [ ]
} ],
"properties" : {
- "owner" : "tamasmate",
- "write.delete.mode" : "merge-on-read"
+ "engine.hive.enabled" : "true",
+ "hive-conf-dir" : "/etc/hive/conf",
+ "write.merge.mode" : "merge-on-read",
+ "connector" : "iceberg",
+ "write.delete.mode" : "merge-on-read",
+ "catalog-database" : "test_db",
+ "write.upsert.enabled" : "true",
+ "catalog-type" : "hive",
+ "write.update.mode" : "merge-on-read",
+ "catalog-name" : "iceberg_hive_catalog"
},
- "current-snapshot-id" : 6816997371555012807,
- "snapshots" : [ {
- "sequence-number" : 1,
- "snapshot-id" : 6816997371555012807,
- "timestamp-ms" : 1649071501670,
- "summary" : {
- "operation" : "append",
- "spark.app.id" : "local-1649071493099",
- "added-data-files" : "1",
- "added-records" : "3",
- "added-files-size" : "662",
- "changed-partition-count" : "1",
- "total-records" : "3",
- "total-files-size" : "662",
- "total-data-files" : "1",
- "total-delete-files" : "0",
- "total-position-deletes" : "0",
- "total-equality-deletes" : "0"
- },
- "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro",
- "schema-id" : 0
- } ],
- "snapshot-log" : [ {
- "timestamp-ms" : 1649071501670,
- "snapshot-id" : 6816997371555012807
- } ],
+ "current-snapshot-id" : -1,
+ "refs" : { },
+ "snapshots" : [ ],
+ "statistics" : [ ],
+ "snapshot-log" : [ ],
"metadata-log" : [ ]
-}
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json
index bed83e8b8..2c278183a 100644
--- a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v2.metadata.json
@@ -1,23 +1,24 @@
{
"format-version" : 2,
- "table-uuid" : "3deb545a-5a19-48f1-ad07-a4d80c677e3e",
+ "table-uuid" : "580ae1f6-35ec-444a-9a34-623227d05f22",
"location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
- "last-sequence-number" : 2,
- "last-updated-ms" : 1649071557501,
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1692789360319,
"last-column-id" : 2,
"current-schema-id" : 0,
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
+ "identifier-field-ids" : [ 1 ],
"fields" : [ {
"id" : 1,
"name" : "id",
- "required" : false,
+ "required" : true,
"type" : "long"
}, {
"id" : 2,
"name" : "data",
- "required" : false,
+ "required" : true,
"type" : "string"
} ]
} ],
@@ -33,61 +34,56 @@
"fields" : [ ]
} ],
"properties" : {
- "owner" : "tamasmate",
- "write.delete.mode" : "merge-on-read"
+ "engine.hive.enabled" : "true",
+ "hive-conf-dir" : "/etc/hive/conf",
+ "write.merge.mode" : "merge-on-read",
+ "connector" : "iceberg",
+ "write.delete.mode" : "merge-on-read",
+ "catalog-database" : "test_db",
+ "write.upsert.enabled" : "true",
+ "catalog-type" : "hive",
+ "write.update.mode" : "merge-on-read",
+ "catalog-name" : "iceberg_hive_catalog"
+ },
+ "current-snapshot-id" : 361598097944440235,
+ "refs" : {
+ "main" : {
+ "snapshot-id" : 361598097944440235,
+ "type" : "branch"
+ }
},
- "current-snapshot-id" : 5725822353600261755,
"snapshots" : [ {
"sequence-number" : 1,
- "snapshot-id" : 6816997371555012807,
- "timestamp-ms" : 1649071501670,
- "summary" : {
- "operation" : "append",
- "spark.app.id" : "local-1649071493099",
- "added-data-files" : "1",
- "added-records" : "3",
- "added-files-size" : "662",
- "changed-partition-count" : "1",
- "total-records" : "3",
- "total-files-size" : "662",
- "total-data-files" : "1",
- "total-delete-files" : "0",
- "total-position-deletes" : "0",
- "total-equality-deletes" : "0"
- },
- "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro",
- "schema-id" : 0
- }, {
- "sequence-number" : 2,
- "snapshot-id" : 5725822353600261755,
- "parent-snapshot-id" : 6816997371555012807,
- "timestamp-ms" : 1649071557501,
+ "snapshot-id" : 361598097944440235,
+ "timestamp-ms" : 1692789360319,
"summary" : {
"operation" : "overwrite",
- "spark.app.id" : "local-1649071493099",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
"added-delete-files" : "1",
- "added-files-size" : "1598",
- "added-position-deletes" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1168",
+ "added-equality-deletes" : "1",
"changed-partition-count" : "1",
- "total-records" : "3",
- "total-files-size" : "2260",
+ "total-records" : "1",
+ "total-files-size" : "1168",
"total-data-files" : "1",
"total-delete-files" : "1",
- "total-position-deletes" : "1",
- "total-equality-deletes" : "0"
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "1"
},
- "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro",
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro",
"schema-id" : 0
} ],
+ "statistics" : [ ],
"snapshot-log" : [ {
- "timestamp-ms" : 1649071501670,
- "snapshot-id" : 6816997371555012807
- }, {
- "timestamp-ms" : 1649071557501,
- "snapshot-id" : 5725822353600261755
+ "timestamp-ms" : 1692789360319,
+ "snapshot-id" : 361598097944440235
} ],
"metadata-log" : [ {
- "timestamp-ms" : 1649071501670,
- "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v1.metadata.json"
+ "timestamp-ms" : 1692789322627,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00000-3ffdeeef-5d15-4a42-81cd-659b9ba984b3.metadata.json"
} ]
-}
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v3.metadata.json
new file mode 100644
index 000000000..1179ea9ec
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v3.metadata.json
@@ -0,0 +1,120 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "580ae1f6-35ec-444a-9a34-623227d05f22",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1692789360703,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "identifier-field-ids" : [ 1 ],
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : true,
+ "type" : "long"
+ }, {
+ "id" : 2,
+ "name" : "data",
+ "required" : true,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "hive-conf-dir" : "/etc/hive/conf",
+ "write.merge.mode" : "merge-on-read",
+ "connector" : "iceberg",
+ "write.delete.mode" : "merge-on-read",
+ "catalog-database" : "test_db",
+ "write.upsert.enabled" : "true",
+ "catalog-type" : "hive",
+ "write.update.mode" : "merge-on-read",
+ "catalog-name" : "iceberg_hive_catalog"
+ },
+ "current-snapshot-id" : 3207673167795000154,
+ "refs" : {
+ "main" : {
+ "snapshot-id" : 3207673167795000154,
+ "type" : "branch"
+ }
+ },
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 361598097944440235,
+ "timestamp-ms" : 1692789360319,
+ "summary" : {
+ "operation" : "overwrite",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
+ "added-delete-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1168",
+ "added-equality-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "1",
+ "total-files-size" : "1168",
+ "total-data-files" : "1",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "1"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 3207673167795000154,
+ "parent-snapshot-id" : 361598097944440235,
+ "timestamp-ms" : 1692789360703,
+ "summary" : {
+ "operation" : "overwrite",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
+ "added-delete-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1166",
+ "added-equality-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "2",
+ "total-files-size" : "2334",
+ "total-data-files" : "2",
+ "total-delete-files" : "2",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "2"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-3207673167795000154-2-30efd05a-a329-4420-8592-9fab221add9e.avro",
+ "schema-id" : 0
+ } ],
+ "statistics" : [ ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1692789360319,
+ "snapshot-id" : 361598097944440235
+ }, {
+ "timestamp-ms" : 1692789360703,
+ "snapshot-id" : 3207673167795000154
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1692789322627,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00000-3ffdeeef-5d15-4a42-81cd-659b9ba984b3.metadata.json"
+ }, {
+ "timestamp-ms" : 1692789360319,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00001-4b107021-78a8-4f79-866e-edda2e44591e.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v4.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v4.metadata.json
new file mode 100644
index 000000000..68f35dd28
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/v4.metadata.json
@@ -0,0 +1,151 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "580ae1f6-35ec-444a-9a34-623227d05f22",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality",
+ "last-sequence-number" : 3,
+ "last-updated-ms" : 1692789361042,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "identifier-field-ids" : [ 1 ],
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : true,
+ "type" : "long"
+ }, {
+ "id" : 2,
+ "name" : "data",
+ "required" : true,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "hive-conf-dir" : "/etc/hive/conf",
+ "write.merge.mode" : "merge-on-read",
+ "connector" : "iceberg",
+ "write.delete.mode" : "merge-on-read",
+ "catalog-database" : "test_db",
+ "write.upsert.enabled" : "true",
+ "catalog-type" : "hive",
+ "write.update.mode" : "merge-on-read",
+ "catalog-name" : "iceberg_hive_catalog"
+ },
+ "current-snapshot-id" : 5763349507283783091,
+ "refs" : {
+ "main" : {
+ "snapshot-id" : 5763349507283783091,
+ "type" : "branch"
+ }
+ },
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 361598097944440235,
+ "timestamp-ms" : 1692789360319,
+ "summary" : {
+ "operation" : "overwrite",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
+ "added-delete-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1168",
+ "added-equality-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "1",
+ "total-files-size" : "1168",
+ "total-data-files" : "1",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "1"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-361598097944440235-1-22029492-dcdb-4ba1-a447-9b9f7200f9e4.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 3207673167795000154,
+ "parent-snapshot-id" : 361598097944440235,
+ "timestamp-ms" : 1692789360703,
+ "summary" : {
+ "operation" : "overwrite",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
+ "added-delete-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1166",
+ "added-equality-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "2",
+ "total-files-size" : "2334",
+ "total-data-files" : "2",
+ "total-delete-files" : "2",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "2"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-3207673167795000154-2-30efd05a-a329-4420-8592-9fab221add9e.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 5763349507283783091,
+ "parent-snapshot-id" : 3207673167795000154,
+ "timestamp-ms" : 1692789361042,
+ "summary" : {
+ "operation" : "overwrite",
+ "flink.job-id" : "a59bb7c5dca5551ff7f1c99e40237364",
+ "flink.max-committed-checkpoint-id" : "9223372036854775807",
+ "added-data-files" : "1",
+ "added-equality-delete-files" : "1",
+ "added-delete-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "1186",
+ "added-equality-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "3520",
+ "total-data-files" : "3",
+ "total-delete-files" : "3",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "3"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/snap-5763349507283783091-3-490632c7-54bf-4362-97e7-b75ab18da9ac.avro",
+ "schema-id" : 0
+ } ],
+ "statistics" : [ ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1692789360319,
+ "snapshot-id" : 361598097944440235
+ }, {
+ "timestamp-ms" : 1692789360703,
+ "snapshot-id" : 3207673167795000154
+ }, {
+ "timestamp-ms" : 1692789361042,
+ "snapshot-id" : 5763349507283783091
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1692789322627,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00000-3ffdeeef-5d15-4a42-81cd-659b9ba984b3.metadata.json"
+ }, {
+ "timestamp-ms" : 1692789360319,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00001-4b107021-78a8-4f79-866e-edda2e44591e.metadata.json"
+ }, {
+ "timestamp-ms" : 1692789360703,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/00002-4853efa4-5e00-474d-9fdb-fa0b27658d63.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text
deleted file mode 100644
index d8263ee98..000000000
--- a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.text
+++ /dev/null
@@ -1 +0,0 @@
-2
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.txt
new file mode 100644
index 000000000..b8626c4cf
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_equality/metadata/version-hint.txt
@@ -0,0 +1 @@
+4
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
index 99ac5d090..9d9cc820e 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
@@ -730,7 +730,7 @@ ImpalaRuntimeException: Iceberg table functional_parquet.iceberg_v2_delete_equal
---- QUERY
# Querying a table with equality deletes is not allowed.
# Use time-travel based on snapshot id.
-select * from functional_parquet.iceberg_v2_delete_equality for system_version as of 5725822353600261755;
+select * from functional_parquet.iceberg_v2_delete_equality for system_version as of 5763349507283783091;
---- CATCH
ImpalaRuntimeException: Iceberg table functional_parquet.iceberg_v2_delete_equality has EQUALITY delete file which is currently not supported by Impala
====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
index b90f02c0b..8f131c864 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
@@ -1162,11 +1162,10 @@ int
====
---- QUERY
# We can query a snapshot if it doesn't have equality deletes.
-select * from iceberg_v2_delete_equality for system_version as of 6816997371555012807;
+select * from iceberg_v2_delete_equality for system_version as of 3207673167795000154;
---- RESULTS
-1,'a'
-2,'b'
-3,'c'
+1,'test_1_base'
+2,'test_2_base'
---- TYPES
BIGINT,STRING
====