You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by bo...@apache.org on 2022/09/02 11:06:11 UTC
[impala] branch master updated (9b4a3ef79 -> d74d6994c)
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 9b4a3ef79 IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
new 85c6366d9 IMPALA-11467: Force Boost to use /dev/random for UUID generation
new cf7490ccb IMPALA-11464: (Addendum) Skip tests in Ozone
new c1bf0af35 IMPALA-11543: Print exception message even if throwable_to_stack_trace_id fails
new 73da4d7dd IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables
new d74d6994c IMPALA-11525: Rename Exec libraries to avoid conflicts with external libraries
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/CMakeLists.txt | 42 +-
be/src/codegen/CMakeLists.txt | 4 +-
be/src/exec/avro/CMakeLists.txt | 10 +-
be/src/exec/hbase/CMakeLists.txt | 6 +-
be/src/exec/kudu/CMakeLists.txt | 6 +-
be/src/exec/orc/CMakeLists.txt | 6 +-
be/src/exec/parquet/CMakeLists.txt | 8 +-
be/src/exec/rcfile/CMakeLists.txt | 6 +-
be/src/exec/sequence/CMakeLists.txt | 6 +-
be/src/exec/text/CMakeLists.txt | 6 +-
be/src/util/jni-util.cc | 2 +
common/thrift/CatalogObjects.thrift | 34 +-
.../analysis/AlterTableSetTblProperties.java | 2 +-
.../java/org/apache/impala/analysis/Analyzer.java | 9 +
.../org/apache/impala/analysis/SelectStmt.java | 8 +-
.../org/apache/impala/catalog/FeCatalogUtils.java | 17 +-
.../java/org/apache/impala/catalog/FeFsTable.java | 61 --
.../org/apache/impala/catalog/FeIcebergTable.java | 101 ++-
.../impala/catalog/IcebergPositionDeleteTable.java | 191 ++++++
.../org/apache/impala/catalog/IcebergTable.java | 4 +
.../org/apache/impala/catalog/VirtualTable.java | 162 +++++
.../impala/catalog/local/LocalIcebergTable.java | 1 -
.../apache/impala/catalog/local/LocalTable.java | 3 +-
.../org/apache/impala/planner/HashJoinNode.java | 3 +-
.../org/apache/impala/planner/HdfsScanNode.java | 54 +-
.../org/apache/impala/planner/IcebergScanNode.java | 400 +-----------
.../apache/impala/planner/IcebergScanPlanner.java | 629 +++++++++++++++++++
.../java/org/apache/impala/planner/JoinNode.java | 6 +-
.../apache/impala/planner/SingleNodePlanner.java | 14 +-
.../java/org/apache/impala/planner/UnionNode.java | 2 +
.../java/org/apache/impala/util/IcebergUtil.java | 72 ++-
.../org/apache/impala/planner/PlannerTest.java | 10 +
testdata/data/README | 105 +++-
...80302-527b-4911-8c6e-88d416adac57-00001.parquet | Bin 1598 -> 1581 bytes
...f065cf59b-job_16597105613620_0031-00001.parquet | Bin 0 -> 625 bytes
.../5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro | Bin 0 -> 3658 bytes
...725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro | Bin 0 -> 2154 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/version-hint.txt | 0
...-67c1e9e1fe29-job_16618645111570_0001-00001.orc | Bin 0 -> 340 bytes
.../a72290c9-c518-4719-8502-6c83a881de07-m0.avro | Bin 0 -> 3654 bytes
...175-1-a72290c9-c518-4719-8502-6c83a881de07.avro | Bin 0 -> 2158 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/version-hint.txt | 0
...c50a84a5e-job_16597105613621_0032-00002.parquet | Bin 0 -> 3222 bytes
...15da206-f60e01cb00000003_1034098606_data.0.parq | Bin 0 -> 1182 bytes
...c50a84a5e-job_16597105613621_0032-00003.parquet | Bin 0 -> 3252 bytes
...15da206-f60e01cb00000003_1489587766_data.0.parq | Bin 0 -> 1203 bytes
...c50a84a5e-job_16597105613621_0032-00001.parquet | Bin 0 -> 3221 bytes
...15da206-f60e01cb00000004_1711435901_data.0.parq | Bin 0 -> 1183 bytes
.../464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro | Bin 0 -> 5766 bytes
.../771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro | Bin 0 -> 4367 bytes
...384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro | Bin 0 -> 2189 bytes
...578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro | Bin 0 -> 2373 bytes
.../metadata/v1.metadata.json | 61 ++
.../metadata/v2.metadata.json | 88 +++
.../metadata/v3.metadata.json | 114 ++++
.../metadata/version-hint.txt | 1 +
...-67bfa860a28c-job_16618645111571_0006-00001.orc | Bin 0 -> 603 bytes
...-1a8309657106-job_16618645111571_0006-00002.orc | Bin 0 -> 1619 bytes
...-67bfa860a28c-job_16618645111571_0006-00002.orc | Bin 0 -> 620 bytes
...-1a8309657106-job_16618645111571_0006-00003.orc | Bin 0 -> 1648 bytes
...-67bfa860a28c-job_16618645111571_0006-00003.orc | Bin 0 -> 584 bytes
...-1a8309657106-job_16618645111571_0006-00001.orc | Bin 0 -> 1615 bytes
.../588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro | Bin 0 -> 4538 bytes
.../5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro | Bin 0 -> 6141 bytes
...310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro | Bin 0 -> 2382 bytes
...108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro | Bin 0 -> 2194 bytes
.../metadata/v1.metadata.json | 61 ++
.../metadata/v2.metadata.json | 86 +++
.../metadata/v3.metadata.json | 112 ++++
.../metadata/version-hint.txt | 1 +
...bea9a4224-job_16597105613620_0025-00001.parquet | Bin 0 -> 625 bytes
...ac53f30b9-job_16597105613621_0025-00001.parquet | Bin 0 -> 2666 bytes
.../236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro | Bin 0 -> 3675 bytes
.../816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro | Bin 0 -> 4159 bytes
...647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro | Bin 0 -> 2335 bytes
...463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro | Bin 0 -> 2171 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/version-hint.txt | 1 +
...-8f7570d3aa7c-job_16618645111570_0002-00001.orc | Bin 0 -> 340 bytes
...-a1cc5b09cf2b-job_16618645111571_0002-00001.orc | Bin 0 -> 1319 bytes
.../6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro | Bin 0 -> 4173 bytes
.../83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro | Bin 0 -> 3671 bytes
...253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro | Bin 0 -> 2343 bytes
...162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro | Bin 0 -> 2175 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/v3.metadata.json | 97 +++
.../metadata/version-hint.txt | 1 +
...9faf82448-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...fa712b987-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...8d0937573-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...89b8d79d3-job_16597105613620_0026-00001.parquet | Bin 0 -> 620 bytes
...89b8d79d3-job_16597105613621_0026-00001.parquet | Bin 0 -> 2697 bytes
...69282a896-job_16597105613621_0027-00001.parquet | Bin 0 -> 2760 bytes
.../4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro | Bin 0 -> 3696 bytes
.../969de65c-8915-4ae5-8d54-a82701195c55-m0.avro | Bin 0 -> 3696 bytes
.../c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro | Bin 0 -> 4222 bytes
.../db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro | Bin 0 -> 3695 bytes
.../ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro | Bin 0 -> 4222 bytes
.../ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro | Bin 0 -> 3696 bytes
...439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro | Bin 0 -> 3119 bytes
...966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro | Bin 0 -> 2191 bytes
...650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro | Bin 0 -> 2377 bytes
...930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro | Bin 0 -> 2749 bytes
...327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro | Bin 0 -> 2563 bytes
...945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro | Bin 0 -> 2934 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/v4.metadata.json | 122 ++++
.../metadata/v5.metadata.json | 148 +++++
.../metadata/v6.metadata.json | 174 ++++++
.../metadata/v7.metadata.json | 200 ++++++
.../metadata/version-hint.txt | 1 +
...-d16898fd4524-job_16618645111570_0003-00001.orc | Bin 0 -> 340 bytes
...-9943cb4e90e9-job_16618645111570_0003-00001.orc | Bin 0 -> 339 bytes
...-28d0bad0a4c0-job_16618645111570_0004-00001.orc | Bin 0 -> 336 bytes
...-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc | Bin 0 -> 341 bytes
...-28d0bad0a4c0-job_16618645111571_0004-00001.orc | Bin 0 -> 1347 bytes
...-0a254d15246c-job_16618645111571_0004-00001.orc | Bin 0 -> 1367 bytes
.../560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro | Bin 0 -> 4237 bytes
.../5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro | Bin 0 -> 3692 bytes
.../666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro | Bin 0 -> 3692 bytes
.../9677edab-12db-429c-aa24-5e3a112caed4-m0.avro | Bin 0 -> 3692 bytes
.../a8152ff6-a39c-435a-916b-d0df02388331-m0.avro | Bin 0 -> 4237 bytes
.../e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro | Bin 0 -> 3692 bytes
...082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro | Bin 0 -> 2956 bytes
...185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro | Bin 0 -> 2196 bytes
...395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro | Bin 0 -> 2576 bytes
...480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro | Bin 0 -> 3146 bytes
...501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro | Bin 0 -> 2386 bytes
...474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro | Bin 0 -> 2766 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/v3.metadata.json | 97 +++
.../metadata/v4.metadata.json | 123 ++++
.../metadata/v5.metadata.json | 149 +++++
.../metadata/v6.metadata.json | 175 ++++++
.../metadata/v7.metadata.json | 201 ++++++
.../metadata/version-hint.txt | 1 +
...f68b8ae1e-job_16597105613620_0030-00001.parquet | Bin 0 -> 625 bytes
...3ae7eaf1f-job_16597105613620_0030-00001.parquet | Bin 0 -> 625 bytes
...3ae7eaf1f-job_16597105613621_0030-00001.parquet | Bin 0 -> 2666 bytes
.../0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro | Bin 0 -> 3675 bytes
.../d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro | Bin 0 -> 3675 bytes
.../da27d446-94df-4850-85dd-c8edcda7685e-m0.avro | Bin 0 -> 4159 bytes
...687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro | Bin 0 -> 2500 bytes
...077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro | Bin 0 -> 2336 bytes
...314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro | Bin 0 -> 2171 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/v4.metadata.json | 122 ++++
.../metadata/version-hint.txt | 1 +
.../functional/functional_schema_template.sql | 137 +++-
.../datasets/functional/schema_constraints.csv | 9 +
.../queries/PlannerTest/iceberg-v2-tables.test | 693 +++++++++++++++++++++
.../queries/QueryTest/iceberg-negative.test | 5 -
.../iceberg-v2-read-position-deletes-orc.test | 201 ++++++
.../iceberg-v2-read-position-deletes.test | 340 ++++++++++
tests/common/skip.py | 2 +
tests/metadata/test_recursive_listing.py | 7 +-
tests/query_test/test_iceberg.py | 29 +-
169 files changed, 6036 insertions(+), 613 deletions(-)
create mode 100644 fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java
create mode 100644 fe/src/main/java/org/apache/impala/catalog/VirtualTable.java
create mode 100644 fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/data/00000-0-data-boroknagyz_20220819180420_a7e5a731-8762-4b59-b3f2-fe6f065cf59b-job_16597105613620_0031-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v2.metadata.json
copy testdata/data/iceberg_test/hadoop_catalog/ice/{airports_orc => iceberg_v2_no_deletes}/metadata/version-hint.txt (100%)
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/data/00000-0-data-boroknagyz_20220830153046_7c9aaa51-d601-4c5b-a51c-67c1e9e1fe29-job_16618645111570_0001-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/a72290c9-c518-4719-8502-6c83a881de07-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/snap-1041485290740594175-1-a72290c9-c518-4719-8502-6c83a881de07.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v2.metadata.json
copy testdata/data/iceberg_test/hadoop_catalog/ice/{airports_orc => iceberg_v2_no_deletes_orc}/metadata/version-hint.txt (100%)
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00002.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/874b32d9a15da206-f60e01cb00000003_1034098606_data.0.parq
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00003.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/874b32d9a15da206-f60e01cb00000003_1489587766_data.0.parq
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/874b32d9a15da206-f60e01cb00000004_1711435901_data.0.parq
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-8885697082976537578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00002.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00002.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00003.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00003.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5359840930115020310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-data-boroknagyz_20220819150927_e95cefe8-5549-42f6-98a6-b1bbea9a4224-job_16597105613620_0025-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-delete-boroknagyz_20220819150939_8f0789df-fcdb-4ba8-875c-4aeac53f30b9-job_16597105613621_0025-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-444149380144800647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-data-boroknagyz_20220830153712_a4060925-000f-4454-b68b-8f7570d3aa7c-job_16618645111570_0002-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-delete-boroknagyz_20220830153733_8f03a46b-e4a5-4643-896b-a1cc5b09cf2b-job_16618645111571_0002-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-1801547319505512253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154646_1cad8c38-c65e-4c7c-b516-c4d9faf82448-job_16597105613620_0026-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154718_db95aeae-d530-4fba-8336-e47fa712b987-job_16597105613620_0026-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154733_162cd69d-80e2-425b-9638-9008d0937573-job_16597105613620_0026-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613620_0026-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613621_0026-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819155000_b0411d64-66e3-49fd-a2f0-dab69282a896-job_16597105613621_0027-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/969de65c-8915-4ae5-8d54-a82701195c55-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-1497619269847778439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v4.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v5.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v6.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v7.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155232_f36b3f3e-183a-475a-a7d1-d16898fd4524-job_16618645111570_0003-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155307_610e0b78-1186-477b-a30b-9943cb4e90e9-job_16618645111570_0003-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111570_0004-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155411_eee97f13-df3d-408c-bcff-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111571_0004-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155430_32c848c9-37ad-4f77-a0fd-0a254d15246c-job_16618645111571_0004-00001.orc
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/9677edab-12db-429c-aa24-5e3a112caed4-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/a8152ff6-a39c-435a-916b-d0df02388331-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8479298697893993480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v4.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v5.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v6.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v7.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/version-hint.txt
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155751_2ddc0706-4c90-42db-927b-d92f68b8ae1e-job_16597105613620_0030-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613620_0030-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-delete-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613621_0030-00001.parquet
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/da27d446-94df-4850-85dd-c8edcda7685e-m0.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-3877007445826010687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v1.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v2.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v3.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v4.metadata.json
create mode 100644 testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/version-hint.txt
create mode 100644 testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test
create mode 100644 testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
create mode 100644 testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
[impala] 02/05: IMPALA-11464: (Addendum) Skip tests in Ozone
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit cf7490ccbdd8ce3ced2a3ae35974de6a7ef91db9
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Wed Aug 31 10:16:27 2022 -0700
IMPALA-11464: (Addendum) Skip tests in Ozone
Updates the skip for new recursive listing tests to match the comment so
that they're only run on HDFS. The previous skip only roughly matched
the set of all non-HDFS filesystems, and didn't automatically include
new filesystems.
Change-Id: I80de83d506138b57a969258b2f6dcf112dd2e44d
Reviewed-on: http://gerrit.cloudera.org:8080/18934
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/common/skip.py | 2 ++
tests/metadata/test_recursive_listing.py | 7 +++----
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/tests/common/skip.py b/tests/common/skip.py
index 70bb0717b..1b4b86643 100644
--- a/tests/common/skip.py
+++ b/tests/common/skip.py
@@ -72,6 +72,8 @@ class SkipIfFS:
hbase = pytest.mark.skipif(not IS_HDFS, reason="HBase not started")
qualified_path = pytest.mark.skipif(not IS_HDFS,
reason="Tests rely on HDFS qualified paths, IMPALA-1872")
+ no_partial_listing = pytest.mark.skipif(not IS_HDFS,
+ reason="Tests rely on HDFS partial listing.")
variable_listing_times = pytest.mark.skipif(IS_S3 or IS_GCS or IS_COS,
reason="Flakiness due to unpredictable listing times on S3.")
eventually_consistent = pytest.mark.skipif(IS_ADLS or IS_COS,
diff --git a/tests/metadata/test_recursive_listing.py b/tests/metadata/test_recursive_listing.py
index 321b78a2d..7b8b0d14b 100644
--- a/tests/metadata/test_recursive_listing.py
+++ b/tests/metadata/test_recursive_listing.py
@@ -141,8 +141,7 @@ class TestRecursiveListing(ImpalaTestSuite):
assert len(self._show_files(fq_tbl_name)) == 1
assert len(self._get_rows(fq_tbl_name)) == 1
- @SkipIfFS.variable_listing_times
- @SkipIfFS.eventually_consistent
+ @SkipIfFS.no_partial_listing
@pytest.mark.execute_serially
@pytest.mark.stress
def test_large_staging_dirs(self, unique_database):
@@ -166,8 +165,7 @@ class TestRecursiveListing(ImpalaTestSuite):
pause_ms_before_file_cleanup=300,
refresh_should_fail=False)
- @SkipIfFS.variable_listing_times
- @SkipIfFS.eventually_consistent
+ @SkipIfFS.no_partial_listing
@pytest.mark.execute_serially
@pytest.mark.stress
def test_partition_dir_removed_inflight(self, unique_database):
@@ -213,6 +211,7 @@ class TestRecursiveListing(ImpalaTestSuite):
handle = self.execute_query_async(refresh_stmt)
# Wait a moment to let REFRESH finish expected partial listing on the dir.
time.sleep(pause_ms_before_file_cleanup / 1000.0)
+ LOG.info("removing staging dir " + large_dir)
self.filesystem_client.delete_file_dir(large_dir[1:], recursive=True)
LOG.info("removed staging dir " + large_dir)
try:
[impala] 04/05: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 73da4d7ddfbcec33c68cd46986b35d7557dd7e67
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Thu Jul 14 15:33:18 2022 +0200
IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables
This patch adds support for reading Iceberg V2 tables use position
deletes. Equality deletes are still not supported. Position delete
files store the file path and file position of the deleted rows.
When an Iceberg table has position delete files we need to do an
ANTI JOIN between data files and delete files. From the data files
we need to query the virtual columns INPUT__FILE__NAME and
FILE__POSITION, while from the delete files we need the data columns
'file_path' and 'pos'. The latter data columns are not part of the
table schema, so we create a virtual table instance of
'IcebergPositionDeleteTable' that has a table schema corresponding
to the delete files ('file_path', 'pos').
This patch introduces a new class 'IcebergScanPlanner' which has
the responsibility of doing a plan for Iceberg table scans. It creates
the aforementioned ANTI JOIN. Also, if there are data files without
corresponding delete files, we can have a separate SCAN node and its
results would be UNIONed to the rows coming from the ANTI JOIN:
UNION
/ \
SCAN data ANTI JOIN
/ \
SCAN data SCAN deletes
Some refactorings in the context of this CR:
Predicate pushdown and time travel logic is transferred from
IcebergScanNode to IcebergScanPlanner. Iceberg snapshot summary
retrieval is moved from FeFsTable to FeIcebergTable.
Testing:
* added planner test
* added e2e tests
TODO in follow-up Jiras:
* better cardinality estimates (IMPALA-11516)
* support unrelative collection columns (select item from t.int_array)
(IMPALA-11517)
Currently such queries return error during analysis
Change-Id: I672cfee18d8e131772d90378d5b12ad4d0f7dd48
Reviewed-on: http://gerrit.cloudera.org:8080/18847
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
common/thrift/CatalogObjects.thrift | 34 +-
.../analysis/AlterTableSetTblProperties.java | 2 +-
.../java/org/apache/impala/analysis/Analyzer.java | 9 +
.../org/apache/impala/analysis/SelectStmt.java | 8 +-
.../org/apache/impala/catalog/FeCatalogUtils.java | 17 +-
.../java/org/apache/impala/catalog/FeFsTable.java | 61 --
.../org/apache/impala/catalog/FeIcebergTable.java | 101 ++-
.../impala/catalog/IcebergPositionDeleteTable.java | 191 ++++++
.../org/apache/impala/catalog/IcebergTable.java | 4 +
.../org/apache/impala/catalog/VirtualTable.java | 162 +++++
.../impala/catalog/local/LocalIcebergTable.java | 1 -
.../apache/impala/catalog/local/LocalTable.java | 3 +-
.../org/apache/impala/planner/HashJoinNode.java | 3 +-
.../org/apache/impala/planner/HdfsScanNode.java | 54 +-
.../org/apache/impala/planner/IcebergScanNode.java | 400 +-----------
.../apache/impala/planner/IcebergScanPlanner.java | 629 +++++++++++++++++++
.../java/org/apache/impala/planner/JoinNode.java | 6 +-
.../apache/impala/planner/SingleNodePlanner.java | 14 +-
.../java/org/apache/impala/planner/UnionNode.java | 2 +
.../java/org/apache/impala/util/IcebergUtil.java | 72 ++-
.../org/apache/impala/planner/PlannerTest.java | 10 +
testdata/data/README | 105 +++-
...80302-527b-4911-8c6e-88d416adac57-00001.parquet | Bin 1598 -> 1581 bytes
...f065cf59b-job_16597105613620_0031-00001.parquet | Bin 0 -> 625 bytes
.../5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro | Bin 0 -> 3658 bytes
...725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro | Bin 0 -> 2154 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/version-hint.txt | 1 +
...-67c1e9e1fe29-job_16618645111570_0001-00001.orc | Bin 0 -> 340 bytes
.../a72290c9-c518-4719-8502-6c83a881de07-m0.avro | Bin 0 -> 3654 bytes
...175-1-a72290c9-c518-4719-8502-6c83a881de07.avro | Bin 0 -> 2158 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/version-hint.txt | 1 +
...c50a84a5e-job_16597105613621_0032-00002.parquet | Bin 0 -> 3222 bytes
...15da206-f60e01cb00000003_1034098606_data.0.parq | Bin 0 -> 1182 bytes
...c50a84a5e-job_16597105613621_0032-00003.parquet | Bin 0 -> 3252 bytes
...15da206-f60e01cb00000003_1489587766_data.0.parq | Bin 0 -> 1203 bytes
...c50a84a5e-job_16597105613621_0032-00001.parquet | Bin 0 -> 3221 bytes
...15da206-f60e01cb00000004_1711435901_data.0.parq | Bin 0 -> 1183 bytes
.../464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro | Bin 0 -> 5766 bytes
.../771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro | Bin 0 -> 4367 bytes
...384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro | Bin 0 -> 2189 bytes
...578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro | Bin 0 -> 2373 bytes
.../metadata/v1.metadata.json | 61 ++
.../metadata/v2.metadata.json | 88 +++
.../metadata/v3.metadata.json | 114 ++++
.../metadata/version-hint.txt | 1 +
...-67bfa860a28c-job_16618645111571_0006-00001.orc | Bin 0 -> 603 bytes
...-1a8309657106-job_16618645111571_0006-00002.orc | Bin 0 -> 1619 bytes
...-67bfa860a28c-job_16618645111571_0006-00002.orc | Bin 0 -> 620 bytes
...-1a8309657106-job_16618645111571_0006-00003.orc | Bin 0 -> 1648 bytes
...-67bfa860a28c-job_16618645111571_0006-00003.orc | Bin 0 -> 584 bytes
...-1a8309657106-job_16618645111571_0006-00001.orc | Bin 0 -> 1615 bytes
.../588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro | Bin 0 -> 4538 bytes
.../5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro | Bin 0 -> 6141 bytes
...310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro | Bin 0 -> 2382 bytes
...108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro | Bin 0 -> 2194 bytes
.../metadata/v1.metadata.json | 61 ++
.../metadata/v2.metadata.json | 86 +++
.../metadata/v3.metadata.json | 112 ++++
.../metadata/version-hint.txt | 1 +
...bea9a4224-job_16597105613620_0025-00001.parquet | Bin 0 -> 625 bytes
...ac53f30b9-job_16597105613621_0025-00001.parquet | Bin 0 -> 2666 bytes
.../236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro | Bin 0 -> 3675 bytes
.../816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro | Bin 0 -> 4159 bytes
...647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro | Bin 0 -> 2335 bytes
...463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro | Bin 0 -> 2171 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/version-hint.txt | 1 +
...-8f7570d3aa7c-job_16618645111570_0002-00001.orc | Bin 0 -> 340 bytes
...-a1cc5b09cf2b-job_16618645111571_0002-00001.orc | Bin 0 -> 1319 bytes
.../6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro | Bin 0 -> 4173 bytes
.../83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro | Bin 0 -> 3671 bytes
...253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro | Bin 0 -> 2343 bytes
...162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro | Bin 0 -> 2175 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/v3.metadata.json | 97 +++
.../metadata/version-hint.txt | 1 +
...9faf82448-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...fa712b987-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...8d0937573-job_16597105613620_0026-00001.parquet | Bin 0 -> 625 bytes
...89b8d79d3-job_16597105613620_0026-00001.parquet | Bin 0 -> 620 bytes
...89b8d79d3-job_16597105613621_0026-00001.parquet | Bin 0 -> 2697 bytes
...69282a896-job_16597105613621_0027-00001.parquet | Bin 0 -> 2760 bytes
.../4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro | Bin 0 -> 3696 bytes
.../969de65c-8915-4ae5-8d54-a82701195c55-m0.avro | Bin 0 -> 3696 bytes
.../c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro | Bin 0 -> 4222 bytes
.../db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro | Bin 0 -> 3695 bytes
.../ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro | Bin 0 -> 4222 bytes
.../ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro | Bin 0 -> 3696 bytes
...439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro | Bin 0 -> 3119 bytes
...966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro | Bin 0 -> 2191 bytes
...650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro | Bin 0 -> 2377 bytes
...930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro | Bin 0 -> 2749 bytes
...327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro | Bin 0 -> 2563 bytes
...945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro | Bin 0 -> 2934 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/v4.metadata.json | 122 ++++
.../metadata/v5.metadata.json | 148 +++++
.../metadata/v6.metadata.json | 174 ++++++
.../metadata/v7.metadata.json | 200 ++++++
.../metadata/version-hint.txt | 1 +
...-d16898fd4524-job_16618645111570_0003-00001.orc | Bin 0 -> 340 bytes
...-9943cb4e90e9-job_16618645111570_0003-00001.orc | Bin 0 -> 339 bytes
...-28d0bad0a4c0-job_16618645111570_0004-00001.orc | Bin 0 -> 336 bytes
...-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc | Bin 0 -> 341 bytes
...-28d0bad0a4c0-job_16618645111571_0004-00001.orc | Bin 0 -> 1347 bytes
...-0a254d15246c-job_16618645111571_0004-00001.orc | Bin 0 -> 1367 bytes
.../560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro | Bin 0 -> 4237 bytes
.../5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro | Bin 0 -> 3692 bytes
.../666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro | Bin 0 -> 3692 bytes
.../9677edab-12db-429c-aa24-5e3a112caed4-m0.avro | Bin 0 -> 3692 bytes
.../a8152ff6-a39c-435a-916b-d0df02388331-m0.avro | Bin 0 -> 4237 bytes
.../e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro | Bin 0 -> 3692 bytes
...082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro | Bin 0 -> 2956 bytes
...185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro | Bin 0 -> 2196 bytes
...395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro | Bin 0 -> 2576 bytes
...480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro | Bin 0 -> 3146 bytes
...501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro | Bin 0 -> 2386 bytes
...474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro | Bin 0 -> 2766 bytes
.../metadata/v1.metadata.json | 46 ++
.../metadata/v2.metadata.json | 71 +++
.../metadata/v3.metadata.json | 97 +++
.../metadata/v4.metadata.json | 123 ++++
.../metadata/v5.metadata.json | 149 +++++
.../metadata/v6.metadata.json | 175 ++++++
.../metadata/v7.metadata.json | 201 ++++++
.../metadata/version-hint.txt | 1 +
...f68b8ae1e-job_16597105613620_0030-00001.parquet | Bin 0 -> 625 bytes
...3ae7eaf1f-job_16597105613620_0030-00001.parquet | Bin 0 -> 625 bytes
...3ae7eaf1f-job_16597105613621_0030-00001.parquet | Bin 0 -> 2666 bytes
.../0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro | Bin 0 -> 3675 bytes
.../d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro | Bin 0 -> 3675 bytes
.../da27d446-94df-4850-85dd-c8edcda7685e-m0.avro | Bin 0 -> 4159 bytes
...687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro | Bin 0 -> 2500 bytes
...077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro | Bin 0 -> 2336 bytes
...314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro | Bin 0 -> 2171 bytes
.../metadata/v1.metadata.json | 45 ++
.../metadata/v2.metadata.json | 70 +++
.../metadata/v3.metadata.json | 96 +++
.../metadata/v4.metadata.json | 122 ++++
.../metadata/version-hint.txt | 1 +
.../functional/functional_schema_template.sql | 137 +++-
.../datasets/functional/schema_constraints.csv | 9 +
.../queries/PlannerTest/iceberg-v2-tables.test | 693 +++++++++++++++++++++
.../queries/QueryTest/iceberg-negative.test | 5 -
.../iceberg-v2-read-position-deletes-orc.test | 201 ++++++
.../iceberg-v2-read-position-deletes.test | 340 ++++++++++
tests/query_test/test_iceberg.py | 29 +-
156 files changed, 5985 insertions(+), 555 deletions(-)
diff --git a/common/thrift/CatalogObjects.thrift b/common/thrift/CatalogObjects.thrift
index 84acf257b..6e3203d77 100644
--- a/common/thrift/CatalogObjects.thrift
+++ b/common/thrift/CatalogObjects.thrift
@@ -259,31 +259,33 @@ struct TColumn {
// Ordinal position in the source table
5: optional i32 position
6: optional TVirtualColumnType virtual_column_type = TVirtualColumnType.NONE
+ // True for hidden columns
+ 7: optional bool is_hidden
// Indicates whether this is an HBase column. If true, implies
// all following HBase-specific fields are set.
- 7: optional bool is_hbase_column
- 8: optional string column_family
- 9: optional string column_qualifier
- 10: optional bool is_binary
+ 8: optional bool is_hbase_column
+ 9: optional string column_family
+ 10: optional string column_qualifier
+ 11: optional bool is_binary
// The followings are Kudu-specific column properties
- 11: optional bool is_kudu_column
- 12: optional bool is_key
- 13: optional bool is_nullable
- 14: optional TColumnEncoding encoding
- 15: optional THdfsCompression compression
- 16: optional Exprs.TExpr default_value
- 17: optional i32 block_size
+ 12: optional bool is_kudu_column
+ 13: optional bool is_key
+ 14: optional bool is_nullable
+ 15: optional TColumnEncoding encoding
+ 16: optional THdfsCompression compression
+ 17: optional Exprs.TExpr default_value
+ 18: optional i32 block_size
// The column name, in the case that it appears in Kudu.
- 18: optional string kudu_column_name
+ 19: optional string kudu_column_name
// Here come the Iceberg-specific fields.
- 19: optional bool is_iceberg_column
- 20: optional i32 iceberg_field_id
+ 20: optional bool is_iceberg_column
+ 21: optional i32 iceberg_field_id
// Key and value field id for Iceberg column with Map type.
- 21: optional i32 iceberg_field_map_key_id
- 22: optional i32 iceberg_field_map_value_id
+ 22: optional i32 iceberg_field_map_key_id
+ 23: optional i32 iceberg_field_map_value_id
}
// Represents an HDFS file in a partition.
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
index 0acd29495..411eca20e 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
@@ -181,7 +181,7 @@ public class AlterTableSetTblProperties extends AlterTableSetStmt {
}
try {
FeIcebergTable iceTable = (FeIcebergTable)getTargetTable();
- List<DataFile> dataFiles = IcebergUtil.getIcebergDataFiles(iceTable,
+ List<DataFile> dataFiles = IcebergUtil.getIcebergFiles(iceTable,
new ArrayList<>(), /*timeTravelSpec=*/null).first;
if (dataFiles.isEmpty()) return;
DataFile firstFile = dataFiles.get(0);
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index a3d50ebfb..25b880f1f 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -63,6 +63,7 @@ import org.apache.impala.catalog.StructType;
import org.apache.impala.catalog.TableLoadingException;
import org.apache.impala.catalog.Type;
import org.apache.impala.catalog.VirtualColumn;
+import org.apache.impala.catalog.VirtualTable;
import org.apache.impala.catalog.local.LocalKuduTable;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.IdGenerator;
@@ -3378,6 +3379,14 @@ public class Analyzer {
return table;
}
+ /**
+ * Adds auxiliary virtual table for a query.
+ */
+ public void addVirtualTable(VirtualTable virtTable) {
+ TableName tblName = virtTable.getTableName();
+ globalState_.stmtTableCache.tables.put(tblName, virtTable);
+ }
+
public org.apache.kudu.client.KuduTable getKuduTable(FeKuduTable feKuduTable)
throws AnalysisException {
String tableName = feKuduTable.getFullName();
diff --git a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
index aad599a2a..8c2b4dcfc 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
@@ -29,14 +29,13 @@ import java.util.Set;
import java.util.stream.Stream;
import java.util.stream.Collectors;
-import org.apache.iceberg.SnapshotSummary;
import org.apache.impala.analysis.Path.PathType;
import org.apache.impala.authorization.Privilege;
import org.apache.impala.catalog.ArrayType;
import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.DatabaseNotFoundException;
-import org.apache.impala.catalog.FeFsTable.Utils;
import org.apache.impala.catalog.FeIcebergTable;
+import org.apache.impala.catalog.FeIcebergTable.Utils;
import org.apache.impala.catalog.FeKuduTable;
import org.apache.impala.catalog.FeTable;
import org.apache.impala.catalog.FeView;
@@ -1359,9 +1358,8 @@ public class SelectStmt extends QueryStmt {
}
if (!hasCountStarFunc) return;
- long num = Utils.getSnapshotSummaryPropOfTypeLong(
- ((FeIcebergTable) table).getIcebergApiTable(), tableRef.getTimeTravelSpec(),
- SnapshotSummary.TOTAL_RECORDS_PROP);
+ long num = Utils.getRecordCount(
+ ((FeIcebergTable) table).getIcebergApiTable(), tableRef.getTimeTravelSpec());
if (num <= 0) return;
analyzer_.setTotalRecordsNum(num);
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java b/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
index 268440b79..a238ff427 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
@@ -98,18 +98,17 @@ public abstract class FeCatalogUtils {
* @throws TableLoadingException if any type is invalid
*/
public static ImmutableList<Column> fieldSchemasToColumns(
- List<FieldSchema> partCols, List<FieldSchema> nonPartCols,
- String tableName, boolean isFullAcidTable) throws TableLoadingException {
+ org.apache.hadoop.hive.metastore.api.Table msTbl) throws TableLoadingException {
+ boolean isFullAcidTable = AcidUtils.isFullAcidTable(msTbl.getParameters());
int pos = 0;
ImmutableList.Builder<Column> ret = ImmutableList.builder();
- for (FieldSchema s : Iterables.concat(partCols, nonPartCols)) {
- if (isFullAcidTable && pos == partCols.size()) {
- ret.add(AcidUtils.getRowIdColumnType(pos));
- ++pos;
+ for (FieldSchema s : Iterables.concat(msTbl.getPartitionKeys(),
+ msTbl.getSd().getCols())) {
+ if (isFullAcidTable && pos == msTbl.getPartitionKeys().size()) {
+ ret.add(AcidUtils.getRowIdColumnType(pos++));
}
- Type type = parseColumnType(s, tableName);
- ret.add(new Column(s.getName(), type, s.getComment(), pos));
- ++pos;
+ Type type = parseColumnType(s, msTbl.getTableName());
+ ret.add(new Column(s.getName(), type, s.getComment(), pos++));
}
return ret.build();
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
index 67980ceed..77b2c7514 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
@@ -34,9 +34,6 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.SQLForeignKey;
import org.apache.hadoop.hive.metastore.api.SQLPrimaryKey;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.util.SnapshotUtil;
import org.apache.impala.analysis.Expr;
import org.apache.impala.analysis.LiteralExpr;
import org.apache.impala.analysis.PartitionKeyValue;
@@ -62,7 +59,6 @@ import org.apache.thrift.TException;
import com.google.common.base.Preconditions;
import com.google.common.base.Joiner;
-import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -471,63 +467,6 @@ public interface FeFsTable extends FeTable {
return result;
}
- /**
- * Get the snapshot summary property of type long from the Iceberg table.
- */
- public static long getSnapshotSummaryPropOfTypeLong(Table icebergTable,
- TimeTravelSpec travelSpec, String propName) {
- String propValue = getSnapshotSummaryProperty(icebergTable, travelSpec, propName);
-
- if (Strings.isNullOrEmpty(propValue)) return -1;
-
- try {
- return Long.parseLong(propValue);
- } catch (NumberFormatException ex) {
- LOG.warn("Failed to get {} from iceberg table summary. Table name: {}, "
- + "Table location: {}, Prop value: {}", propName, icebergTable.name(),
- icebergTable.location(), propValue, ex);
- }
-
- return -1;
- }
-
- /**
- * Get the snapshot summary property from the Iceberg table.
- */
- private static String getSnapshotSummaryProperty(Table icebergTable,
- TimeTravelSpec travelSpec, String propName) {
- Snapshot snapshot = getIcebergSnapshot(icebergTable, travelSpec);
- // There are no snapshots for the tables created for the first time.
- if (snapshot == null) { return null; }
- return snapshot.summary().get(propName);
- }
-
- /**
- * Get time-travel snapshot or current snapshot of the Iceberg table.
- * Only current snapshot can return null.
- */
- private static Snapshot getIcebergSnapshot(Table icebergTable,
- TimeTravelSpec travelSpec) {
-
- if (travelSpec == null) return icebergTable.currentSnapshot();
-
- Snapshot snapshot;
- if (travelSpec.getKind().equals(Kind.VERSION_AS_OF)) {
- long snapshotId = travelSpec.getAsOfVersion();
- snapshot = icebergTable.snapshot(snapshotId);
- Preconditions.checkArgument(snapshot != null, "Cannot find snapshot with ID %s",
- snapshotId);
- } else {
- long timestampMillis = travelSpec.getAsOfMillis();
- long snapshotId = SnapshotUtil.snapshotIdAsOfTime(icebergTable, timestampMillis);
- snapshot = icebergTable.snapshot(snapshotId);
- Preconditions.checkArgument(snapshot != null,
- "Cannot find snapshot with ID %s, timestampMillis %s", snapshotId,
- timestampMillis);
- }
- return snapshot;
- }
-
/**
* Get file info for the given fe iceberg table.
*/
diff --git a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
index 6ab595d8f..bea82e370 100644
--- a/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
@@ -35,17 +35,25 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.iceberg.ContentFile;
import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.PartitionField;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotSummary;
import org.apache.iceberg.Table;
+import org.apache.iceberg.util.SnapshotUtil;
import org.apache.impala.analysis.IcebergPartitionField;
import org.apache.impala.analysis.IcebergPartitionSpec;
import org.apache.impala.analysis.LiteralExpr;
import org.apache.impala.analysis.PartitionKeyValue;
+import org.apache.impala.analysis.TimeTravelSpec;
+import org.apache.impala.analysis.TimeTravelSpec.Kind;
import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
import org.apache.impala.common.FileSystemUtil;
+import org.apache.impala.common.Pair;
import org.apache.impala.common.Reference;
import org.apache.impala.compat.HdfsShim;
import org.apache.impala.thrift.TColumn;
@@ -67,6 +75,8 @@ import org.apache.impala.util.TResultRowBuilder;
import org.json.simple.JSONValue;
import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.Iterables;
import com.google.common.primitives.Ints;
import org.slf4j.Logger;
@@ -546,26 +556,27 @@ public interface FeIcebergTable extends FeFsTable {
}
}
Map<String, HdfsPartition.FileDescriptor> fileDescMap = new HashMap<>();
- List<DataFile> dataFileList = IcebergUtil.getIcebergDataFiles(table,
- new ArrayList<>(), /*timeTravelSpecl=*/null).first;
- for (DataFile dataFile : dataFileList) {
- Path path = new Path(dataFile.path().toString());
+ Pair<List<DataFile>, Set<DeleteFile>> allFiles = IcebergUtil.getIcebergFiles(
+ table, new ArrayList<>(), /*timeTravelSpecl=*/null);
+ for (ContentFile contentFile : Iterables.concat(allFiles.first,
+ allFiles.second)) {
+ Path path = new Path(contentFile.path().toString());
if (hdfsFileDescMap.containsKey(path.toUri().getPath())) {
- String pathHash = IcebergUtil.getDataFilePathHash(dataFile);
+ String pathHash = IcebergUtil.getFilePathHash(contentFile);
HdfsPartition.FileDescriptor fsFd = hdfsFileDescMap.get(
path.toUri().getPath());
HdfsPartition.FileDescriptor iceFd = fsFd.cloneWithFileMetadata(
- IcebergUtil.createIcebergMetadata(table, dataFile));
+ IcebergUtil.createIcebergMetadata(table, contentFile));
fileDescMap.put(pathHash, iceFd);
} else {
- LOG.warn("Iceberg DataFile '{}' cannot be found in the HDFS recursive file "
+ LOG.warn("Iceberg file '{}' cannot be found in the HDFS recursive file "
+ "listing results.", path.toString());
HdfsPartition.FileDescriptor fileDesc = getFileDescriptor(
- new Path(dataFile.path().toString()),
+ new Path(contentFile.path().toString()),
new Path(table.getIcebergTableLocation()), table.getHostIndex());
HdfsPartition.FileDescriptor iceFd = fileDesc.cloneWithFileMetadata(
- IcebergUtil.createIcebergMetadata(table, dataFile));
- fileDescMap.put(IcebergUtil.getDataFilePathHash(dataFile), iceFd);
+ IcebergUtil.createIcebergMetadata(table, contentFile));
+ fileDescMap.put(IcebergUtil.getFilePathHash(contentFile), iceFd);
}
}
return fileDescMap;
@@ -575,12 +586,13 @@ public interface FeIcebergTable extends FeFsTable {
* Return the PartitionContent loaded by the internal HdfsTable. To avoid returning
* the metadata files the result set is limited to the files that are tracked by
* Iceberg. Both the number of rows and number of files show in partitionStats.
+ * TODO(IMPALA-11516): Return better partition stats for V2 tables.
*/
public static Map<String, TIcebergPartitionStats> loadPartitionStats(
IcebergTable table) throws TableLoadingException {
List<DataFile> dataFileList =
IcebergUtil
- .getIcebergDataFiles(table, new ArrayList<>(), /*timeTravelSpecl=*/null)
+ .getIcebergFiles(table, new ArrayList<>(), /*timeTravelSpecl=*/null)
.first;
Map<String, TIcebergPartitionStats> partitionStats = new HashMap<>();
for (DataFile dataFile : dataFileList) {
@@ -664,5 +676,72 @@ public interface FeIcebergTable extends FeFsTable {
IcebergUtil.toTHdfsFileFormat(icebergTable.getIcebergFileFormat()));
}
}
+
+ /**
+ * Get record count for a table with the given time travel spec. Returns -1 when
+ * the record count cannot be retrieved from the table summary.
+ * If 'travelSpec' is null then the current snapshot is being used.
+ */
+ public static long getRecordCount(Table icebergTable,
+ TimeTravelSpec travelSpec) {
+ Map<String, String> summary = getSnapshotSummary(icebergTable, travelSpec);
+ if (summary == null) return -1;
+
+ String totalRecordsStr = summary.get(SnapshotSummary.TOTAL_RECORDS_PROP);
+ if (Strings.isNullOrEmpty(totalRecordsStr)) return -1;
+
+ try {
+ // We cannot tell the record count from the summary if there are deleted rows.
+ String totalDeleteFilesStr = summary.get(SnapshotSummary.TOTAL_DELETE_FILES_PROP);
+ if (!Strings.isNullOrEmpty(totalDeleteFilesStr)) {
+ long totalDeleteFiles = Long.parseLong(totalDeleteFilesStr);
+ if (totalDeleteFiles > 0) return -1;
+ }
+
+ return Long.parseLong(totalRecordsStr);
+ } catch (NumberFormatException ex) {
+ LOG.warn("Failed to get {} from iceberg table summary. Table name: {}, " +
+ "Table location: {}, Prop value: {}",
+ SnapshotSummary.TOTAL_RECORDS_PROP, icebergTable.name(),
+ icebergTable.location(), totalRecordsStr, ex);
+ }
+
+ return -1;
+ }
+
+ /**
+ * Get the snapshot summary from the Iceberg table.
+ */
+ private static Map<String, String> getSnapshotSummary(Table icebergTable,
+ TimeTravelSpec travelSpec) {
+ Snapshot snapshot = getIcebergSnapshot(icebergTable, travelSpec);
+ // There are no snapshots for the tables created for the first time.
+ if (snapshot == null) return null;
+ return snapshot.summary();
+ }
+
+ /**
+ * Get time-travel snapshot or current snapshot of the Iceberg table.
+ * Only current snapshot can return null.
+ */
+ private static Snapshot getIcebergSnapshot(Table icebergTable,
+ TimeTravelSpec travelSpec) {
+ if (travelSpec == null) return icebergTable.currentSnapshot();
+ Snapshot snapshot;
+ if (travelSpec.getKind().equals(Kind.VERSION_AS_OF)) {
+ long snapshotId = travelSpec.getAsOfVersion();
+ snapshot = icebergTable.snapshot(snapshotId);
+ Preconditions.checkArgument(snapshot != null, "Cannot find snapshot with ID %s",
+ snapshotId);
+ } else {
+ long timestampMillis = travelSpec.getAsOfMillis();
+ long snapshotId = SnapshotUtil.snapshotIdAsOfTime(icebergTable, timestampMillis);
+ snapshot = icebergTable.snapshot(snapshotId);
+ Preconditions.checkArgument(snapshot != null,
+ "Cannot find snapshot with ID %s, timestampMillis %s", snapshotId,
+ timestampMillis);
+ }
+ return snapshot;
+ }
}
}
diff --git a/fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java b/fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java
new file mode 100644
index 000000000..da917bc74
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java
@@ -0,0 +1,191 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.catalog;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.hive.common.ValidWriteIdList;
+import org.apache.iceberg.Table;
+import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
+import org.apache.impala.analysis.IcebergPartitionSpec;
+import org.apache.impala.analysis.TableName;
+import org.apache.impala.thrift.CatalogObjectsConstants;
+import org.apache.impala.thrift.TCatalogObjectType;
+import org.apache.impala.thrift.TColumnStats;
+import org.apache.impala.thrift.TCompressionCodec;
+import org.apache.impala.thrift.THdfsPartition;
+import org.apache.impala.thrift.THdfsTable;
+import org.apache.impala.thrift.TIcebergCatalog;
+import org.apache.impala.thrift.TIcebergFileFormat;
+import org.apache.impala.thrift.TIcebergPartitionStats;
+import org.apache.impala.thrift.TTableDescriptor;
+import org.apache.impala.thrift.TTableStats;
+import org.apache.impala.thrift.TTableType;
+import org.apache.impala.util.AcidUtils;
+import org.apache.impala.util.IcebergUtil;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Iceberg position delete table is created on the fly during planning. It belongs to an
+ * actual Iceberg table (referred to as 'baseTable_'), but has a schema that corresponds
+ * to the file schema of position delete files. Therefore with the help of it we can
+ * do an ANTI JOIN between data files and delete files.
+ */
+public class IcebergPositionDeleteTable extends VirtualTable implements FeIcebergTable {
+ private FeIcebergTable baseTable_;
+ private Set<FileDescriptor> deleteFiles_;
+ private long deleteRecordsCount_;
+
+ public static String FILE_PATH_COLUMN = "file_path";
+ public static String POS_COLUMN = "pos";
+
+ public IcebergPositionDeleteTable(FeIcebergTable baseTable, String name,
+ Set<FileDescriptor> deleteFiles,
+ long deleteRecordsCount, TColumnStats filePathsStats) {
+ super(baseTable.getMetaStoreTable(), baseTable.getDb(), name,
+ baseTable.getOwnerUser());
+ baseTable_ = baseTable;
+ deleteFiles_ = deleteFiles;
+ deleteRecordsCount_ = deleteRecordsCount;
+ Column filePath = new IcebergColumn(FILE_PATH_COLUMN, Type.STRING, /*comment=*/"",
+ colsByPos_.size(), IcebergTable.V2_FILE_PATH_FIELD_ID, -1, -1,
+ /*nullable=*/false);
+ Column pos = new IcebergColumn(POS_COLUMN, Type.BIGINT, /*comment=*/"",
+ colsByPos_.size(), IcebergTable.V2_POS_FIELD_ID, -1, -1, /*nullable=*/false);
+ filePath.updateStats(filePathsStats);
+ pos.updateStats(getPosStats(pos));
+ addColumn(filePath);
+ addColumn(pos);
+ }
+
+ private TColumnStats getPosStats(Column pos) {
+ TColumnStats colStats = new TColumnStats();
+ colStats.num_distinct_values = deleteRecordsCount_;
+ colStats.num_nulls = 0;
+ colStats.max_size = pos.getType().getSlotSize();
+ return colStats;
+ }
+
+ @Override
+ public long getNumRows() {
+ return deleteRecordsCount_;
+ }
+
+ @Override
+ public TTableStats getTTableStats() {
+ long totalBytes = 0;
+ for (FileDescriptor df : deleteFiles_) {
+ totalBytes += df.getFileLength();
+ }
+ TTableStats ret = new TTableStats(getNumRows());
+ ret.setTotal_file_bytes(totalBytes);
+ return ret;
+ }
+
+ /**
+ * Return same descriptor as the base table, but with a schema that corresponds to
+ * the position delete file schema ('file_path', 'pos').
+ */
+ @Override
+ public TTableDescriptor toThriftDescriptor(int tableId,
+ Set<Long> referencedPartitions) {
+ TTableDescriptor desc = baseTable_.toThriftDescriptor(tableId, referencedPartitions);
+ desc.setColumnDescriptors(FeCatalogUtils.getTColumnDescriptors(this));
+ return desc;
+ }
+
+ @Override
+ public Map<String, FileDescriptor> getPathHashToFileDescMap() {
+ return baseTable_.getPathHashToFileDescMap();
+ }
+
+ @Override
+ public Map<String, TIcebergPartitionStats> getIcebergPartitionStats() {
+ return null;
+ }
+
+ @Override
+ public FeFsTable getFeFsTable() {
+ return baseTable_.getFeFsTable();
+ }
+
+ @Override
+ public TIcebergCatalog getIcebergCatalog() {
+ return null;
+ }
+
+ @Override
+ public Table getIcebergApiTable() {
+ return baseTable_.getIcebergApiTable();
+ }
+
+ @Override
+ public String getIcebergCatalogLocation() {
+ return null;
+ }
+
+ @Override
+ public TIcebergFileFormat getIcebergFileFormat() {
+ return baseTable_.getIcebergFileFormat();
+ }
+
+ @Override
+ public TCompressionCodec getIcebergParquetCompressionCodec() {
+ return null;
+ }
+
+ @Override
+ public long getIcebergParquetRowGroupSize() {
+ return baseTable_.getIcebergParquetRowGroupSize();
+ }
+
+ @Override
+ public long getIcebergParquetPlainPageSize() {
+ return baseTable_.getIcebergParquetPlainPageSize();
+ }
+
+ @Override
+ public long getIcebergParquetDictPageSize() {
+ return baseTable_.getIcebergParquetDictPageSize();
+ }
+
+ @Override
+ public String getIcebergTableLocation() {
+ return null;
+ }
+
+ @Override
+ public List<IcebergPartitionSpec> getPartitionSpecs() {
+ return null;
+ }
+
+ @Override
+ public IcebergPartitionSpec getDefaultPartitionSpec() {
+ return null;
+ }
+
+ @Override
+ public int getDefaultPartitionSpecId() {
+ return -1;
+ }
+}
diff --git a/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
index 2cb36930a..2b36bfa55 100644
--- a/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
@@ -125,6 +125,10 @@ public class IcebergTable extends Table implements FeIcebergTable {
public static final long MIN_PARQUET_PAGE_SIZE = 64 * 1024;
public static final long MAX_PARQUET_PAGE_SIZE = 1024 * 1024 * 1024;
+ // Field IDs of the position delete files according to the Iceberg spec.
+ public static final int V2_FILE_PATH_FIELD_ID = 2147483546;
+ public static final int V2_POS_FIELD_ID = 2147483545;
+
// Iceberg catalog type dependend on table properties
private TIcebergCatalog icebergCatalog_;
diff --git a/fe/src/main/java/org/apache/impala/catalog/VirtualTable.java b/fe/src/main/java/org/apache/impala/catalog/VirtualTable.java
new file mode 100644
index 000000000..1df7632f3
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/catalog/VirtualTable.java
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.catalog;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import org.apache.hadoop.hive.common.ValidWriteIdList;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.impala.analysis.TableName;
+import org.apache.impala.thrift.TCatalogObjectType;
+import org.apache.impala.thrift.TImpalaTableType;
+import org.apache.impala.util.AcidUtils;
+
+/**
+ * Tables that aren't actually exist in HMS. E.g. Iceberg position delete tables.
+ * Metadata tables can be another use case for this class.
+ * Using old table schema and stats during time-travel might be another use case.
+ */
+public abstract class VirtualTable implements FeTable {
+ protected final Table msTable_;
+ protected final FeDb db_;
+ protected final String name_;
+ protected final String owner_;
+
+ // colsByPos[i] refers to the ith column in the table. The first numClusteringCols are
+ // the clustering columns.
+ protected final List<Column> colsByPos_ = new ArrayList<>();
+
+ // map from lowercase column name to Column object.
+ protected final Map<String, Column> colsByName_ = new HashMap<>();
+
+ // Number of clustering columns.
+ protected int numClusteringCols_ = 0;
+
+ // Type of this table (array of struct) that mirrors the columns. Useful for analysis.
+ protected final ArrayType type_ = new ArrayType(new StructType());
+
+ public VirtualTable(org.apache.hadoop.hive.metastore.api.Table msTable, FeDb db,
+ String name, String owner) {
+ msTable_ = msTable;
+ db_ = db;
+ name_ = name;
+ owner_ = owner;
+ }
+
+ protected void addColumn(Column col) {
+ colsByPos_.add(col);
+ colsByName_.put(col.getName().toLowerCase(), col);
+ ((StructType) type_.getItemType()).addField(
+ new StructField(col.getName(), col.getType(), col.getComment()));
+ }
+
+ @Override
+ public String getTableComment() { return null; }
+
+ @Override
+ public boolean isLoaded() { return true; }
+
+ @Override
+ public Table getMetaStoreTable() { return msTable_; }
+
+ @Override
+ public String getStorageHandlerClassName() { return null; }
+
+ @Override
+ public TCatalogObjectType getCatalogObjectType() { return TCatalogObjectType.TABLE; }
+
+ @Override
+ public TImpalaTableType getTableType() { return TImpalaTableType.TABLE; }
+
+ @Override
+ public FeDb getDb() { return db_; }
+
+ @Override
+ public String getName() { return name_; }
+
+ @Override
+ public String getFullName() { return (db_ != null ? db_.getName() + "." : "") + name_; }
+
+ @Override
+ public TableName getTableName() {
+ return new TableName(db_ != null ? db_.getName() : null, name_);
+ }
+
+ @Override
+ public List<Column> getColumns() { return colsByPos_; }
+
+ @Override
+ public List<Column> getColumnsInHiveOrder() {
+ List<Column> columns = Lists.newArrayList(getNonClusteringColumns());
+ if (getMetaStoreTable() != null &&
+ AcidUtils.isFullAcidTable(getMetaStoreTable().getParameters())) {
+ // Remove synthetic "row__id" column.
+ Preconditions.checkState(columns.get(0).getName().equals("row__id"));
+ columns.remove(0);
+ }
+ columns.addAll(getClusteringColumns());
+ return Collections.unmodifiableList(columns);
+ }
+
+ @Override
+ public List<Column> getClusteringColumns() {
+ return Collections.unmodifiableList(colsByPos_.subList(0, numClusteringCols_));
+ }
+
+ @Override
+ public List<Column> getNonClusteringColumns() {
+ return Collections.unmodifiableList(colsByPos_.subList(numClusteringCols_,
+ colsByPos_.size()));
+ }
+
+ @Override
+ public List<String> getColumnNames() { return Column.toColumnNames(colsByPos_); }
+
+ @Override
+ public SqlConstraints getSqlConstraints() { return null; }
+
+ @Override
+ public int getNumClusteringCols() { return numClusteringCols_; }
+
+ @Override
+ public boolean isClusteringColumn(Column c) {
+ return c.getPosition() < numClusteringCols_;
+ }
+
+ @Override // FeTable
+ public Column getColumn(String name) { return colsByName_.get(name.toLowerCase()); }
+
+ @Override
+ public ArrayType getType() { return type_; }
+
+ @Override
+ public long getWriteId() { return 0; }
+
+ @Override
+ public ValidWriteIdList getValidWriteIds() { return null; }
+
+ @Override
+ public String getOwnerUser() { return owner_; }
+}
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
index 84f1bf13a..0853b1396 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
@@ -34,7 +34,6 @@ import org.apache.impala.catalog.FeFsTable;
import org.apache.impala.catalog.FeIcebergTable;
import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
import org.apache.impala.catalog.TableLoadingException;
-import org.apache.impala.catalog.VirtualColumn;
import org.apache.impala.thrift.TCompressionCodec;
import org.apache.impala.thrift.THdfsPartition;
import org.apache.impala.thrift.THdfsTable;
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
index 3b131217a..5e56cd9e9 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
@@ -348,9 +348,8 @@ abstract class LocalTable implements FeTable {
// then all other columns.
List<Column> cols;
try {
+ cols = FeCatalogUtils.fieldSchemasToColumns(msTbl);
boolean isFullAcidTable = AcidUtils.isFullAcidTable(msTbl.getParameters());
- cols = FeCatalogUtils.fieldSchemasToColumns(msTbl.getPartitionKeys(),
- msTbl.getSd().getCols(), msTbl.getTableName(), isFullAcidTable);
return new ColumnMap(cols, numClusteringCols, fullName, isFullAcidTable);
} catch (TableLoadingException e) {
throw new LocalCatalogException(e);
diff --git a/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java b/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
index 254c242d1..0fe91eed3 100644
--- a/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
@@ -175,7 +175,8 @@ public class HashJoinNode extends JoinNode {
}
}
if (detailLevel.ordinal() > TExplainLevel.MINIMAL.ordinal()) {
- if (!isAcidJoin_ || detailLevel.ordinal() >= TExplainLevel.EXTENDED.ordinal()) {
+ if (!isDeleteRowsJoin_ ||
+ detailLevel.ordinal() >= TExplainLevel.EXTENDED.ordinal()) {
output.append(detailPrefix + "hash predicates: ");
for (int i = 0; i < eqJoinConjuncts_.size(); ++i) {
Expr eqConjunct = eqJoinConjuncts_.get(i);
diff --git a/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java b/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
index 68994ffad..227911426 100644
--- a/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
@@ -201,6 +201,7 @@ public class HdfsScanNode extends ScanNode {
.add(HdfsFileFormat.PARQUET)
.add(HdfsFileFormat.HUDI_PARQUET)
.add(HdfsFileFormat.ORC)
+ .add(HdfsFileFormat.ICEBERG)
.build();
//An estimate of the width of a row when the information is not available.
@@ -1185,28 +1186,11 @@ public class HdfsScanNode extends ScanNode {
throw new ImpalaRuntimeException("Error determining partition fs type", e);
}
boolean fsHasBlocks = FileSystemUtil.supportsStorageIds(partitionFs);
- List<FileDescriptor> fileDescs;
- if (this instanceof IcebergScanNode) {
- fileDescs = ((IcebergScanNode) this).getFileDescriptorByIcebergPredicates();
- } else {
- if (isSimpleLimit) {
- fileDescs = new ArrayList<>();
- for (FileDescriptor fd : partition.getFileDescriptors()) {
- // skip empty files
- if ((fsHasBlocks && fd.getNumFileBlocks() == 0)
- || (!fsHasBlocks && fd.getFileLength() <= 0)) {
- continue;
- }
- simpleLimitNumRows++; // conservatively estimate 1 row per file
- fileDescs.add(fd);
- if (simpleLimitNumRows == analyzer.getSimpleLimitStatus().second) {
- break;
- }
- }
- } else {
- fileDescs = partition.getFileDescriptors();
- }
- }
+ List<FileDescriptor> fileDescs = getFileDescriptorsWithLimit(partition, fsHasBlocks,
+ isSimpleLimit ? analyzer.getSimpleLimitStatus().second - simpleLimitNumRows
+ : -1);
+ // conservatively estimate 1 row per file
+ simpleLimitNumRows += fileDescs.size();
if (sampledFiles != null) {
// If we are sampling, check whether this partition is included in the sample.
@@ -1297,6 +1281,30 @@ public class HdfsScanNode extends ScanNode {
}
}
+ protected List<FileDescriptor> getFileDescriptorsWithLimit(
+ FeFsPartition partition, boolean fsHasBlocks, long limit) {
+ List<FileDescriptor> fileDescs;
+ if (limit != -1) {
+ int fileCount = 0;
+ fileDescs = new ArrayList<>();
+ for (FileDescriptor fd : partition.getFileDescriptors()) {
+ // skip empty files
+ if ((fsHasBlocks && fd.getNumFileBlocks() == 0)
+ || (!fsHasBlocks && fd.getFileLength() <= 0)) {
+ continue;
+ }
+ fileCount++;
+ fileDescs.add(fd);
+ if (fileCount == limit) {
+ break;
+ }
+ }
+ } else {
+ fileDescs = partition.getFileDescriptors();
+ }
+ return fileDescs;
+ }
+
/**
* Update the estimate of maximum number of rows per scan range based on the fraction
* of bytes of the scan range relative to the total bytes per partition or table.
@@ -1652,7 +1660,7 @@ public class HdfsScanNode extends ScanNode {
* ESTIMATED_COMPRESSION_FACTOR_LEGACY);
} else {
Preconditions.checkState(VALID_COLUMNAR_FORMATS.contains(format),
- "Unknown HDFS compressed format: %s", this);
+ "Unknown HDFS compressed format: %s", format, this);
estimatedPartitionSize += Math.round(p.getSize()
* ESTIMATED_COMPRESSION_FACTOR_COLUMNAR);
}
diff --git a/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java b/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
index d065bc723..5085e48fb 100644
--- a/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
@@ -17,80 +17,43 @@
package org.apache.impala.planner;
-import java.io.IOException;
-import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
-import org.apache.hadoop.fs.Path;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.Schema;
-import org.apache.iceberg.expressions.Expression;
-import org.apache.iceberg.expressions.Expressions;
-import org.apache.iceberg.expressions.Expression.Operation;
-import org.apache.iceberg.expressions.UnboundPredicate;
-import org.apache.iceberg.types.Types;
-import org.apache.impala.analysis.Analyzer;
-import org.apache.impala.analysis.BinaryPredicate;
-import org.apache.impala.analysis.BoolLiteral;
-import org.apache.impala.analysis.CompoundPredicate;
-import org.apache.impala.analysis.DateLiteral;
import org.apache.impala.analysis.Expr;
-import org.apache.impala.analysis.InPredicate;
-import org.apache.impala.analysis.IsNullPredicate;
-import org.apache.impala.analysis.LiteralExpr;
import org.apache.impala.analysis.MultiAggregateInfo;
-import org.apache.impala.analysis.NumericLiteral;
-import org.apache.impala.analysis.SlotRef;
-import org.apache.impala.analysis.StringLiteral;
import org.apache.impala.analysis.TableRef;
-import org.apache.impala.analysis.TimeTravelSpec;
-import org.apache.impala.analysis.TupleDescriptor;
-import org.apache.impala.catalog.Column;
import org.apache.impala.catalog.FeCatalogUtils;
import org.apache.impala.catalog.FeFsPartition;
import org.apache.impala.catalog.FeFsTable;
import org.apache.impala.catalog.FeIcebergTable;
-import org.apache.impala.catalog.IcebergColumn;
-import org.apache.impala.catalog.TableLoadingException;
-import org.apache.impala.catalog.Type;
import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
-import org.apache.impala.common.AnalysisException;
-import org.apache.impala.common.ImpalaException;
-import org.apache.impala.common.ImpalaRuntimeException;
-import org.apache.impala.common.InternalException;
-import org.apache.impala.common.Pair;
-import org.apache.impala.util.IcebergUtil;
-import org.apache.impala.util.ExprUtil;
import com.google.common.base.Preconditions;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
/**
- * Scan of a single iceberg table
+ * Scan of a single iceberg table.
*/
public class IcebergScanNode extends HdfsScanNode {
- private final static Logger LOG = LoggerFactory.getLogger(TimeTravelSpec.class);
-
- private final FeIcebergTable icebergTable_;
- // Exprs in icebergConjuncts_ converted to Expression.
- private final List<Expression> icebergPredicates_ = new ArrayList<>();
+ private List<FileDescriptor> fileDescs_;
- private TimeTravelSpec timeTravelSpec_;
-
- public IcebergScanNode(PlanNodeId id, TupleDescriptor desc, List<Expr> conjuncts,
- TableRef tblRef, FeFsTable feFsTable, MultiAggregateInfo aggInfo) {
- super(id, desc, conjuncts, getIcebergPartition(feFsTable), tblRef, aggInfo,
- null, false);
- icebergTable_ = (FeIcebergTable) desc_.getTable();
- timeTravelSpec_ = tblRef.getTimeTravelSpec();
+ public IcebergScanNode(PlanNodeId id, TableRef tblRef, List<Expr> conjuncts,
+ MultiAggregateInfo aggInfo, List<FileDescriptor> fileDescs) {
+ super(id, tblRef.getDesc(), conjuncts,
+ getIcebergPartition(((FeIcebergTable)tblRef.getTable()).getFeFsTable()), tblRef,
+ aggInfo, null, false);
// Hdfs table transformed from iceberg table only has one partition
Preconditions.checkState(partitions_.size() == 1);
+ fileDescs_ = fileDescs;
+ }
+
+ /**
+ * In some cases we exactly know the cardinality, e.g. POSITION DELETE scan node.
+ */
+ public void setCardinality(long cardinality) {
+ cardinality_ = cardinality;
}
/**
@@ -104,330 +67,19 @@ public class IcebergScanNode extends HdfsScanNode {
}
@Override
- public void init(Analyzer analyzer) throws ImpalaException {
- extractIcebergConjuncts(analyzer);
- super.init(analyzer);
- }
-
- /**
- * We need prune hdfs partition FileDescriptor by iceberg predicates
- */
- public List<FileDescriptor> getFileDescriptorByIcebergPredicates()
- throws ImpalaRuntimeException {
- List<DataFile> dataFileList;
- try {
- Pair<List<DataFile>, Boolean> dataFileListAndDeletePair =
- IcebergUtil.getIcebergDataFiles(icebergTable_, icebergPredicates_,
- timeTravelSpec_);
- dataFileList = dataFileListAndDeletePair.first;
- Boolean hasDeleteFile = dataFileListAndDeletePair.second;
- if (hasDeleteFile) {
- throw new TableLoadingException(String.format("Unsupported Iceberg V2 feature, "
- + "table '%s' with snapshot id '%s' contains delete files.",
- icebergTable_.getFullName(), icebergTable_.snapshotId()));
+ protected List<FileDescriptor> getFileDescriptorsWithLimit(
+ FeFsPartition partition, boolean fsHasBlocks, long limit) {
+ if (limit != -1) {
+ long cnt = 0;
+ List<FileDescriptor> ret = new ArrayList<>();
+ for (FileDescriptor fd : fileDescs_) {
+ if (cnt == limit) break;
+ ret.add(fd);
+ ++cnt;
}
- } catch (TableLoadingException e) {
- throw new ImpalaRuntimeException(String.format(
- "Failed to load data files for Iceberg table: %s", icebergTable_.getFullName()),
- e);
- }
- long dataFilesCacheMisses = 0;
- List<FileDescriptor> fileDescList = new ArrayList<>();
- for (DataFile dataFile : dataFileList) {
- FileDescriptor fileDesc = icebergTable_.getPathHashToFileDescMap()
- .get(IcebergUtil.getDataFilePathHash(dataFile));
- if (fileDesc == null) {
- if (timeTravelSpec_ == null) {
- // We should always find the data files in the cache when not doing time travel.
- throw new ImpalaRuntimeException("Cannot find file in cache: " + dataFile.path()
- + " with snapshot id: " + String.valueOf(icebergTable_.snapshotId()));
- }
- ++dataFilesCacheMisses;
- try {
- fileDesc = FeIcebergTable.Utils.getFileDescriptor(
- new Path(dataFile.path().toString()),
- new Path(icebergTable_.getIcebergTableLocation()),
- icebergTable_.getHostIndex());
- } catch (IOException ex) {
- throw new ImpalaRuntimeException(
- "Cannot load file descriptor for " + dataFile.path(), ex);
- }
- if (fileDesc == null) {
- throw new ImpalaRuntimeException(
- "Cannot load file descriptor for: " + dataFile.path());
- }
- // Add file descriptor to the cache.
- fileDesc = fileDesc.cloneWithFileMetadata(
- IcebergUtil.createIcebergMetadata(icebergTable_, dataFile));
- icebergTable_.getPathHashToFileDescMap().put(
- IcebergUtil.getDataFilePathHash(dataFile), fileDesc);
- }
- fileDescList.add(fileDesc);
- }
-
- if (dataFilesCacheMisses > 0) {
- Preconditions.checkState(timeTravelSpec_ != null);
- LOG.info("File descriptors had to be loaded on demand during time travel: " +
- String.valueOf(dataFilesCacheMisses));
- }
-
- return fileDescList;
- }
-
- /**
- * Extracts predicates from conjuncts_ that can be pushed down to Iceberg.
- *
- * Since Iceberg will filter data files by metadata instead of scan data files,
- * we pushdown all predicates to Iceberg to get the minimum data files to scan.
- * Here are three cases for predicate pushdown:
- * 1.The column is not part of any Iceberg partition expression
- * 2.The column is part of all partition keys without any transformation (i.e. IDENTITY)
- * 3.The column is part of all partition keys with transformation (i.e. MONTH/DAY/HOUR)
- * We can use case 1 and 3 to filter data files, but also need to evaluate it in the
- * scan, for case 2 we don't need to evaluate it in the scan. So we evaluate all
- * predicates in the scan to keep consistency. More details about Iceberg scanning,
- * please refer: https://iceberg.apache.org/spec/#scan-planning
- */
- private void extractIcebergConjuncts(Analyzer analyzer) throws ImpalaException {
- for (Expr expr : conjuncts_) {
- tryConvertIcebergPredicate(analyzer, expr);
- }
- }
-
- /**
- * Returns Iceberg operator by BinaryPredicate operator, or null if the operation
- * is not supported by Iceberg.
- */
- private Operation getIcebergOperator(BinaryPredicate.Operator op) {
- switch (op) {
- case EQ: return Operation.EQ;
- case NE: return Operation.NOT_EQ;
- case LE: return Operation.LT_EQ;
- case GE: return Operation.GT_EQ;
- case LT: return Operation.LT;
- case GT: return Operation.GT;
- default: return null;
- }
- }
-
- /**
- * Returns Iceberg operator by CompoundPredicate operator, or null if the operation
- * is not supported by Iceberg.
- */
- private Operation getIcebergOperator(CompoundPredicate.Operator op) {
- switch (op) {
- case AND: return Operation.AND;
- case OR: return Operation.OR;
- case NOT: return Operation.NOT;
- default: return null;
- }
- }
-
- /**
- * Transform impala predicate to iceberg predicate
- */
- private void tryConvertIcebergPredicate(Analyzer analyzer, Expr expr)
- throws ImpalaException {
- Expression predicate = convertIcebergPredicate(analyzer, expr);
- if (predicate != null) {
- icebergPredicates_.add(predicate);
- LOG.debug("Push down the predicate: " + predicate + " to iceberg");
- }
- }
-
- private Expression convertIcebergPredicate(Analyzer analyzer, Expr expr)
- throws ImpalaException {
- if (expr instanceof BinaryPredicate) {
- return convertIcebergPredicate(analyzer, (BinaryPredicate) expr);
- } else if (expr instanceof InPredicate) {
- return convertIcebergPredicate(analyzer, (InPredicate) expr);
- } else if (expr instanceof IsNullPredicate) {
- return convertIcebergPredicate((IsNullPredicate) expr);
- } else if (expr instanceof CompoundPredicate) {
- return convertIcebergPredicate(analyzer, (CompoundPredicate) expr);
+ return ret;
} else {
- return null;
- }
- }
-
- private UnboundPredicate<Object> convertIcebergPredicate(Analyzer analyzer,
- BinaryPredicate predicate) throws ImpalaException {
- Operation op = getIcebergOperator(predicate.getOp());
- if (op == null) {
- return null;
- }
-
- // Do not convert if there is an implicit cast
- if (!(predicate.getChild(0) instanceof SlotRef)) {
- return null;
- }
- SlotRef ref = (SlotRef) predicate.getChild(0);
-
- if (!(predicate.getChild(1) instanceof LiteralExpr)) {
- return null;
- }
- LiteralExpr literal = (LiteralExpr) predicate.getChild(1);
-
- // If predicate contains map/struct, this column would be null
- Column col = ref.getDesc().getColumn();
- if (col == null) {
- return null;
- }
-
- Object value = getIcebergValue(analyzer, ref, literal);
- if (value == null) {
- return null;
- }
-
- return Expressions.predicate(op, col.getName(), value);
- }
-
- private UnboundPredicate<Object> convertIcebergPredicate(Analyzer analyzer,
- InPredicate predicate) throws ImpalaException {
- // Do not convert if there is an implicit cast
- if (!(predicate.getChild(0) instanceof SlotRef)) {
- return null;
- }
- SlotRef ref = (SlotRef) predicate.getChild(0);
-
- // If predicate contains map/struct, this column would be null
- Column col = ref.getDesc().getColumn();
- if (col == null) {
- return null;
- }
-
- // Expressions takes a list of values as Objects
- List<Object> values = new ArrayList<>();
- for (int i = 1; i < predicate.getChildren().size(); ++i) {
- if (!Expr.IS_LITERAL.apply(predicate.getChild(i))) {
- return null;
- }
- LiteralExpr literal = (LiteralExpr) predicate.getChild(i);
-
- // Cannot push IN or NOT_IN predicate with null literal values
- if (Expr.IS_NULL_LITERAL.apply(literal)) {
- return null;
- }
-
- Object value = getIcebergValue(analyzer, ref, literal);
- if (value == null) {
- return null;
- }
-
- values.add(value);
- }
-
- // According to the method:
- // 'org.apache.iceberg.expressions.InclusiveMetricsEvaluator.MetricsEvalVisitor#notIn'
- // Expressions.notIn only works when the push-down column is the partition column
- if (predicate.isNotIn())
- return Expressions.notIn(col.getName(), values);
- else {
- return Expressions.in(col.getName(), values);
- }
- }
-
- private UnboundPredicate<Object> convertIcebergPredicate(IsNullPredicate predicate) {
- // Do not convert if there is an implicit cast
- if (!(predicate.getChild(0) instanceof SlotRef)) {
- return null;
- }
- SlotRef ref = (SlotRef) predicate.getChild(0);
-
- // If predicate contains map/struct, this column would be null
- Column col = ref.getDesc().getColumn();
- if (col == null) {
- return null;
- }
-
- if (predicate.isNotNull()) {
- return Expressions.notNull(col.getName());
- } else{
- return Expressions.isNull(col.getName());
- }
- }
-
- private Expression convertIcebergPredicate(Analyzer analyzer,
- CompoundPredicate predicate) throws ImpalaException {
- Operation op = getIcebergOperator(predicate.getOp());
- if (op == null) {
- return null;
- }
-
- Expression left = convertIcebergPredicate(analyzer, predicate.getChild(0));
- if (left == null) {
- return null;
- }
- if (op.equals(Operation.NOT)) {
- return Expressions.not(left);
- }
-
- Expression right = convertIcebergPredicate(analyzer, predicate.getChild(1));
- if (right == null) {
- return null;
- }
- return op.equals(Operation.AND) ? Expressions.and(left, right)
- : Expressions.or(left, right);
- }
-
- private Object getIcebergValue(Analyzer analyzer, SlotRef ref, LiteralExpr literal)
- throws ImpalaException {
- IcebergColumn iceCol = (IcebergColumn) ref.getDesc().getColumn();
- Schema iceSchema = icebergTable_.getIcebergSchema();
- switch (literal.getType().getPrimitiveType()) {
- case BOOLEAN: return ((BoolLiteral) literal).getValue();
- case TINYINT:
- case SMALLINT:
- case INT: return ((NumericLiteral) literal).getIntValue();
- case BIGINT: return ((NumericLiteral) literal).getLongValue();
- case FLOAT: return (float) ((NumericLiteral) literal).getDoubleValue();
- case DOUBLE: return ((NumericLiteral) literal).getDoubleValue();
- case STRING:
- case DATETIME:
- case CHAR: return ((StringLiteral) literal).getUnescapedValue();
- case TIMESTAMP: return getIcebergTsValue(analyzer, literal, iceCol, iceSchema);
- case DATE: return ((DateLiteral) literal).getValue();
- case DECIMAL: return getIcebergDecimalValue(ref, (NumericLiteral) literal);
- default: {
- Preconditions.checkState(false,
- "Unsupported iceberg type considered for predicate: %s",
- literal.getType().toSql());
- }
- }
- return null;
- }
-
- private BigDecimal getIcebergDecimalValue(SlotRef ref, NumericLiteral literal) {
- Type colType = ref.getDesc().getColumn().getType();
- int scale = colType.getDecimalDigits();
- BigDecimal literalValue = literal.getValue();
-
- if (literalValue.scale() > scale) return null;
- // Iceberg DecimalLiteral needs to have the exact same scale.
- if (literalValue.scale() < scale) return literalValue.setScale(scale);
- return literalValue;
- }
-
- private Object getIcebergTsValue(Analyzer analyzer, LiteralExpr literal,
- IcebergColumn iceCol, Schema iceSchema) throws AnalysisException {
- try {
- org.apache.iceberg.types.Type iceType = iceSchema.findType(iceCol.getFieldId());
- Preconditions.checkState(iceType instanceof Types.TimestampType);
- Types.TimestampType tsType = (Types.TimestampType) iceType;
- if (tsType.shouldAdjustToUTC()) {
- return ExprUtil.localTimestampToUnixTimeMicros(analyzer, literal);
- } else {
- return ExprUtil.utcTimestampToUnixTimeMicros(analyzer, literal);
- }
- } catch (InternalException ex) {
- // We cannot interpret the timestamp literal. Maybe the timestamp is invalid,
- // or the local timestamp ambigously converts to UTC due to daylight saving
- // time backward turn. E.g. '2021-10-31 02:15:00 Europe/Budapest' converts to
- // either '2021-10-31 00:15:00 UTC' or '2021-10-31 01:15:00 UTC'.
- LOG.warn("Exception occurred during timestamp conversion: " + ex.toString() +
- "\nThis means timestamp predicate is not pushed to Iceberg, let Impala " +
- "backend handle it.");
+ return fileDescs_;
}
- return null;
}
}
diff --git a/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
new file mode 100644
index 000000000..a974d9322
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.planner;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import org.apache.curator.shaded.com.google.common.collect.Lists;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Types;
+import org.apache.impala.analysis.Analyzer;
+import org.apache.impala.analysis.BinaryPredicate;
+import org.apache.impala.analysis.CompoundPredicate;
+import org.apache.impala.analysis.Expr;
+import org.apache.impala.analysis.InPredicate;
+import org.apache.impala.analysis.IsNullPredicate;
+import org.apache.impala.analysis.JoinOperator;
+import org.apache.impala.analysis.LiteralExpr;
+import org.apache.impala.catalog.Column;
+import org.apache.impala.catalog.FeFsTable;
+import org.apache.impala.analysis.BoolLiteral;
+import org.apache.impala.analysis.DateLiteral;
+import org.apache.impala.analysis.MultiAggregateInfo;
+import org.apache.impala.analysis.NumericLiteral;
+import org.apache.impala.analysis.SlotDescriptor;
+import org.apache.impala.analysis.SlotRef;
+import org.apache.impala.analysis.StringLiteral;
+import org.apache.impala.analysis.TableRef;
+import org.apache.impala.analysis.TimeTravelSpec;
+import org.apache.impala.analysis.TupleDescriptor;
+import org.apache.impala.analysis.BinaryPredicate.Operator;
+import org.apache.impala.catalog.FeIcebergTable;
+import org.apache.impala.catalog.FeTable;
+import org.apache.impala.catalog.IcebergColumn;
+import org.apache.impala.catalog.IcebergPositionDeleteTable;
+import org.apache.impala.catalog.IcebergTable;
+import org.apache.impala.catalog.TableLoadingException;
+import org.apache.impala.catalog.Type;
+import org.apache.impala.catalog.VirtualColumn;
+import org.apache.impala.catalog.HdfsPartition.FileDescriptor;
+import org.apache.impala.common.AnalysisException;
+import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.ImpalaRuntimeException;
+import org.apache.impala.common.InternalException;
+import org.apache.impala.common.Pair;
+import org.apache.impala.planner.JoinNode.DistributionMode;
+import org.apache.impala.thrift.TColumnStats;
+import org.apache.impala.thrift.TVirtualColumnType;
+import org.apache.impala.util.ExprUtil;
+import org.apache.impala.util.IcebergUtil;
+
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergScanPlanner {
+ private final static Logger LOG = LoggerFactory.getLogger(IcebergScanPlanner.class);
+
+ private Analyzer analyzer_;
+ private PlannerContext ctx_;
+ private TableRef tblRef_;
+ private List<Expr> conjuncts_;
+ private MultiAggregateInfo aggInfo_;
+
+ // Exprs in icebergConjuncts_ converted to Expression.
+ private final List<Expression> icebergPredicates_ = new ArrayList<>();
+
+ private List<FileDescriptor> dataFilesWithoutDeletes_ = new ArrayList<>();
+ private List<FileDescriptor> dataFilesWithDeletes_ = new ArrayList<>();
+ private Set<FileDescriptor> deleteFiles_ = new HashSet<>();
+
+ // Statistics about the data and delete files. Useful for memory estimates of the
+ // ANTI JOIN
+ private long deletesRecordCount_ = 0;
+ private long dataFilesWithDeletesSumPaths_ = 0;
+ private long dataFilesWithDeletesMaxPath_ = 0;
+
+ public IcebergScanPlanner(Analyzer analyzer, PlannerContext ctx,
+ TableRef iceTblRef, List<Expr> conjuncts, MultiAggregateInfo aggInfo)
+ throws ImpalaException {
+ Preconditions.checkState(iceTblRef.getTable() instanceof FeIcebergTable);
+ analyzer_ = analyzer;
+ ctx_ = ctx;
+ tblRef_ = iceTblRef;
+ conjuncts_ = conjuncts;
+ aggInfo_ = aggInfo;
+ extractIcebergConjuncts();
+ }
+
+ public PlanNode createIcebergScanPlan() throws ImpalaException {
+ FeIcebergTable iceTbl = getIceTable();
+ analyzer_.materializeSlots(conjuncts_);
+ filterFileDescriptors();
+
+ PlanNode ret;
+ if (deleteFiles_.isEmpty()) {
+ // If there are no delete files we can just create a single SCAN node.
+ Preconditions.checkState(dataFilesWithDeletes_.isEmpty());
+ ret = new IcebergScanNode(ctx_.getNextNodeId(), tblRef_, conjuncts_,
+ aggInfo_, dataFilesWithoutDeletes_);
+ ret.init(analyzer_);
+ } else {
+ // Let's create a bit more complex plan in the presence of delete files.
+ ret = createComplexIcebergScanPlan();
+ }
+ return ret;
+ }
+
+ private PlanNode createComplexIcebergScanPlan() throws ImpalaException {
+ PlanNode joinNode = createPositionJoinNode();
+ if (dataFilesWithoutDeletes_.isEmpty()) {
+ // All data files has corresponding delete files, so we just return an ANTI JOIN
+ // between all data files and all delete files.
+ return joinNode;
+ }
+ // If there are data files without corresponding delete files to be applied, we
+ // can just create a SCAN node for these and do a UNION ALL with the ANTI JOIN.
+ IcebergScanNode dataScanNode = new IcebergScanNode(
+ ctx_.getNextNodeId(), tblRef_, conjuncts_, aggInfo_, dataFilesWithoutDeletes_);
+ dataScanNode.init(analyzer_);
+ List<Expr> outputExprs = tblRef_.getDesc().getSlots().stream().map(
+ entry -> new SlotRef(entry)).collect(Collectors.toList());
+ UnionNode unionNode = new UnionNode(ctx_.getNextNodeId(), tblRef_.getId(),
+ outputExprs, false);
+ unionNode.addChild(dataScanNode, outputExprs);
+ unionNode.addChild(joinNode, outputExprs);
+ unionNode.init(analyzer_);
+ // Verify that the children are passed through.
+ Preconditions.checkState(unionNode.getChildCount() == 2);
+ Preconditions.checkState(unionNode.getFirstNonPassthroughChildIndex() == 2);
+ return unionNode;
+ }
+
+ private PlanNode createPositionJoinNode() throws ImpalaException {
+ // The followings just create separate scan nodes for data files and position delete
+ // files, plus adds a LEFT ANTI HASH JOIN above them.
+ PlanNodeId dataScanNodeId = ctx_.getNextNodeId();
+ PlanNodeId deleteScanNodeId = ctx_.getNextNodeId();
+ IcebergPositionDeleteTable deleteTable = new IcebergPositionDeleteTable(getIceTable(),
+ getIceTable().getName() + "-POSITION-DELETE-" + deleteScanNodeId.toString(),
+ deleteFiles_, deletesRecordCount_, getFilePathStats());
+ analyzer_.addVirtualTable(deleteTable);
+ TableRef deleteDeltaRef = TableRef.newTableRef(analyzer_,
+ Arrays.asList(deleteTable.getDb().getName(), deleteTable.getName()),
+ tblRef_.getUniqueAlias() + "-position-delete");
+ addDataVirtualPositionSlots(tblRef_);
+ addDeletePositionSlots(deleteDeltaRef);
+ IcebergScanNode dataScanNode = new IcebergScanNode(
+ dataScanNodeId, tblRef_, conjuncts_, aggInfo_, dataFilesWithDeletes_);
+ dataScanNode.init(analyzer_);
+ IcebergScanNode deleteScanNode = new IcebergScanNode(
+ deleteScanNodeId, deleteDeltaRef, /*conjuncts=*/Collections.emptyList(),
+ aggInfo_, Lists.newArrayList(deleteFiles_));
+ deleteScanNode.init(analyzer_);
+ deleteScanNode.setCardinality(deletesRecordCount_);
+
+ // Now let's create the JOIN node
+ List<BinaryPredicate> positionJoinConjuncts = createPositionJoinConjuncts(
+ analyzer_, tblRef_.getDesc(), deleteDeltaRef.getDesc());
+
+ JoinNode joinNode = new HashJoinNode(dataScanNode, deleteScanNode,
+ /*straight_join=*/true, DistributionMode.NONE, JoinOperator.LEFT_ANTI_JOIN,
+ positionJoinConjuncts, /*otherJoinConjuncts=*/Collections.emptyList());
+ joinNode.setId(ctx_.getNextNodeId());
+ joinNode.init(analyzer_);
+ joinNode.setIsDeleteRowsJoin();
+ return joinNode;
+ }
+
+ private void addDataVirtualPositionSlots(TableRef tblRef) throws AnalysisException {
+ List<String> rawPath = new ArrayList<>();
+ rawPath.add(tblRef.getUniqueAlias());
+ // Add slot refs for position delete fields;
+ String[] posFields = {VirtualColumn.INPUT_FILE_NAME.getName(),
+ VirtualColumn.FILE_POSITION.getName()};
+ for (String posField : posFields) {
+ rawPath.add(posField);
+ SingleNodePlanner.addSlotRefToDesc(analyzer_, rawPath);
+ rawPath.remove(rawPath.size() - 1);
+ }
+ }
+
+ private void addDeletePositionSlots(TableRef tblRef)
+ throws AnalysisException {
+ List<String> rawPath = new ArrayList<>();
+ rawPath.add(tblRef.getUniqueAlias());
+ // Add slot refs for position delete fields;
+ String[] posFields = {IcebergPositionDeleteTable.FILE_PATH_COLUMN,
+ IcebergPositionDeleteTable.POS_COLUMN};
+ for (String posField : posFields) {
+ rawPath.add(posField);
+ SingleNodePlanner.addSlotRefToDesc(analyzer_, rawPath);
+ rawPath.remove(rawPath.size() - 1);
+ }
+ }
+
+ private List<BinaryPredicate> createPositionJoinConjuncts(Analyzer analyzer,
+ TupleDescriptor insertTupleDesc, TupleDescriptor deleteTupleDesc)
+ throws AnalysisException {
+ List<BinaryPredicate> ret = new ArrayList<>();
+ for (SlotDescriptor deleteSlotDesc : deleteTupleDesc.getSlots()) {
+ boolean foundMatch = false;
+ Column col = deleteSlotDesc.getParent().getTable().getColumns().get(
+ deleteSlotDesc.getMaterializedPath().get(0));
+ Preconditions.checkState(col instanceof IcebergColumn);
+ int fieldId = ((IcebergColumn)col).getFieldId();
+ for (SlotDescriptor insertSlotDesc : insertTupleDesc.getSlots()) {
+ TVirtualColumnType virtColType = insertSlotDesc.getVirtualColumnType();
+ if (fieldId == IcebergTable.V2_FILE_PATH_FIELD_ID &&
+ virtColType != TVirtualColumnType.INPUT_FILE_NAME) {
+ continue;
+ }
+ if (fieldId == IcebergTable.V2_POS_FIELD_ID &&
+ virtColType != TVirtualColumnType.FILE_POSITION) {
+ continue;
+ }
+ foundMatch = true;
+ BinaryPredicate pred = new BinaryPredicate(
+ Operator.EQ, new SlotRef(insertSlotDesc), new SlotRef(deleteSlotDesc));
+ pred.analyze(analyzer);
+ ret.add(pred);
+ break;
+ }
+ Preconditions.checkState(foundMatch);
+ }
+ return ret;
+ }
+
+ private void filterFileDescriptors() throws ImpalaException {
+ TimeTravelSpec timeTravelSpec = tblRef_.getTimeTravelSpec();
+
+ try (CloseableIterable<FileScanTask> fileScanTasks = IcebergUtil.planFiles(
+ getIceTable(), icebergPredicates_, timeTravelSpec)) {
+ long dataFilesCacheMisses = 0;
+ for (FileScanTask fileScanTask : fileScanTasks) {
+ Pair<FileDescriptor, Boolean> fileDesc = getFileDescriptor(fileScanTask.file());
+ if (!fileDesc.second) ++dataFilesCacheMisses;
+ if (fileScanTask.deletes().isEmpty()) {
+ dataFilesWithoutDeletes_.add(fileDesc.first);
+ } else {
+ updateDataFilesWithDeletesStatistics(fileScanTask.file());
+ dataFilesWithDeletes_.add(fileDesc.first);
+ for (DeleteFile delFile : fileScanTask.deletes()) {
+ // TODO(IMPALA-11388): Add support for equality deletes.
+ if (delFile.content() == FileContent.EQUALITY_DELETES) {
+ throw new ImpalaRuntimeException(String.format(
+ "Iceberg table %s has EQUALITY delete file which is currently " +
+ "not supported by Impala: %s", getIceTable().getFullName(),
+ delFile.path()));
+ }
+ Pair<FileDescriptor, Boolean> delFileDesc = getFileDescriptor(delFile);
+ if (!delFileDesc.second) ++dataFilesCacheMisses;
+ if (deleteFiles_.add(delFileDesc.first)) {
+ // New delete file added to 'deleteFiles_'.
+ updateDeleteFilesStatistics(delFile);
+ }
+ }
+ }
+ }
+
+ if (dataFilesCacheMisses > 0) {
+ Preconditions.checkState(timeTravelSpec != null);
+ LOG.info("File descriptors had to be loaded on demand during time travel: " +
+ String.valueOf(dataFilesCacheMisses));
+ }
+ } catch (IOException | TableLoadingException e) {
+ throw new ImpalaRuntimeException(String.format(
+ "Failed to load data files for Iceberg table: %s", getIceTable().getFullName()),
+ e);
+ }
+ }
+
+ private void updateDataFilesWithDeletesStatistics(ContentFile cf) {
+ long pathSize = cf.path().toString().length();
+ dataFilesWithDeletesSumPaths_ += pathSize;
+ if (pathSize > dataFilesWithDeletesMaxPath_) {
+ dataFilesWithDeletesMaxPath_ = pathSize;
+ }
+ }
+
+ private void updateDeleteFilesStatistics(DeleteFile delFile) {
+ deletesRecordCount_ += getRecordCount(delFile);
+ }
+
+ private long getRecordCount(DeleteFile delFile) {
+ long recordCount = delFile.recordCount();
+ // 'record_count' is a required field for Iceberg data files, but let's still
+ // prepare for the case when a compute engine doesn't fill it.
+ if (recordCount == -1) return 1000;
+ return recordCount;
+ }
+
+ private FeIcebergTable getIceTable() { return (FeIcebergTable)tblRef_.getTable(); }
+
+ private TColumnStats getFilePathStats() {
+ TColumnStats colStats = new TColumnStats();
+ colStats.avg_size = dataFilesWithDeletesSumPaths_ / dataFilesWithDeletes_.size();
+ colStats.max_size = dataFilesWithDeletesMaxPath_;
+ colStats.num_distinct_values = dataFilesWithDeletes_.size();
+ colStats.num_nulls = 0;
+ return colStats;
+ }
+
+ private Pair<FileDescriptor, Boolean> getFileDescriptor(ContentFile cf)
+ throws ImpalaRuntimeException {
+ boolean cachehit = true;
+ FileDescriptor fileDesc = getIceTable().getPathHashToFileDescMap().get(
+ IcebergUtil.getFilePathHash(cf));
+ if (fileDesc == null) {
+ if (tblRef_.getTimeTravelSpec() == null) {
+ // We should always find the data files in the cache when not doing time travel.
+ throw new ImpalaRuntimeException("Cannot find file in cache: " + cf.path()
+ + " with snapshot id: " + String.valueOf(getIceTable().snapshotId()));
+ }
+ cachehit = false;
+ try {
+ fileDesc = FeIcebergTable.Utils.getFileDescriptor(
+ new Path(cf.path().toString()),
+ new Path(getIceTable().getIcebergTableLocation()),
+ getIceTable().getHostIndex());
+ } catch (IOException ex) {
+ throw new ImpalaRuntimeException(
+ "Cannot load file descriptor for " + cf.path(), ex);
+ }
+ if (fileDesc == null) {
+ throw new ImpalaRuntimeException(
+ "Cannot load file descriptor for: " + cf.path());
+ }
+ // Add file descriptor to the cache.
+ fileDesc = fileDesc.cloneWithFileMetadata(
+ IcebergUtil.createIcebergMetadata(getIceTable(), cf));
+ getIceTable().getPathHashToFileDescMap().put(
+ IcebergUtil.getFilePathHash(cf), fileDesc);
+ }
+ return new Pair<>(fileDesc, cachehit);
+ }
+
+ /**
+ * Extracts predicates from conjuncts_ that can be pushed down to Iceberg.
+ *
+ * Since Iceberg will filter data files by metadata instead of scan data files,
+ * we pushdown all predicates to Iceberg to get the minimum data files to scan.
+ * Here are three cases for predicate pushdown:
+ * 1.The column is not part of any Iceberg partition expression
+ * 2.The column is part of all partition keys without any transformation (i.e. IDENTITY)
+ * 3.The column is part of all partition keys with transformation (i.e. MONTH/DAY/HOUR)
+ * We can use case 1 and 3 to filter data files, but also need to evaluate it in the
+ * scan, for case 2 we don't need to evaluate it in the scan. So we evaluate all
+ * predicates in the scan to keep consistency. More details about Iceberg scanning,
+ * please refer: https://iceberg.apache.org/spec/#scan-planning
+ */
+ private void extractIcebergConjuncts() throws ImpalaException {
+ for (Expr expr : conjuncts_) {
+ tryConvertIcebergPredicate(expr);
+ }
+ }
+
+ /**
+ * Returns Iceberg operator by BinaryPredicate operator, or null if the operation
+ * is not supported by Iceberg.
+ */
+ private Operation getIcebergOperator(BinaryPredicate.Operator op) {
+ switch (op) {
+ case EQ: return Operation.EQ;
+ case NE: return Operation.NOT_EQ;
+ case LE: return Operation.LT_EQ;
+ case GE: return Operation.GT_EQ;
+ case LT: return Operation.LT;
+ case GT: return Operation.GT;
+ default: return null;
+ }
+ }
+
+ /**
+ * Returns Iceberg operator by CompoundPredicate operator, or null if the operation
+ * is not supported by Iceberg.
+ */
+ private Operation getIcebergOperator(CompoundPredicate.Operator op) {
+ switch (op) {
+ case AND: return Operation.AND;
+ case OR: return Operation.OR;
+ case NOT: return Operation.NOT;
+ default: return null;
+ }
+ }
+
+ /**
+ * Transform impala predicate to iceberg predicate
+ */
+ private void tryConvertIcebergPredicate(Expr expr)
+ throws ImpalaException {
+ Expression predicate = convertIcebergPredicate(expr);
+ if (predicate != null) {
+ icebergPredicates_.add(predicate);
+ LOG.debug("Push down the predicate: " + predicate + " to iceberg");
+ }
+ }
+
+ private Expression convertIcebergPredicate(Expr expr)
+ throws ImpalaException {
+ if (expr instanceof BinaryPredicate) {
+ return convertIcebergPredicate((BinaryPredicate) expr);
+ } else if (expr instanceof InPredicate) {
+ return convertIcebergPredicate((InPredicate) expr);
+ } else if (expr instanceof IsNullPredicate) {
+ return convertIcebergPredicate((IsNullPredicate) expr);
+ } else if (expr instanceof CompoundPredicate) {
+ return convertIcebergPredicate((CompoundPredicate) expr);
+ } else {
+ return null;
+ }
+ }
+
+ private UnboundPredicate<Object> convertIcebergPredicate(BinaryPredicate predicate)
+ throws ImpalaException {
+ Operation op = getIcebergOperator(predicate.getOp());
+ if (op == null) {
+ return null;
+ }
+
+ // Do not convert if there is an implicit cast
+ if (!(predicate.getChild(0) instanceof SlotRef)) {
+ return null;
+ }
+ SlotRef ref = (SlotRef) predicate.getChild(0);
+
+ if (!(predicate.getChild(1) instanceof LiteralExpr)) {
+ return null;
+ }
+ LiteralExpr literal = (LiteralExpr) predicate.getChild(1);
+
+ // If predicate contains map/struct, this column would be null
+ Column col = ref.getDesc().getColumn();
+ if (col == null) {
+ return null;
+ }
+
+ Object value = getIcebergValue(ref, literal);
+ if (value == null) {
+ return null;
+ }
+
+ return Expressions.predicate(op, col.getName(), value);
+ }
+
+ private UnboundPredicate<Object> convertIcebergPredicate(InPredicate predicate)
+ throws ImpalaException {
+ // Do not convert if there is an implicit cast
+ if (!(predicate.getChild(0) instanceof SlotRef)) {
+ return null;
+ }
+ SlotRef ref = (SlotRef) predicate.getChild(0);
+
+ // If predicate contains map/struct, this column would be null
+ Column col = ref.getDesc().getColumn();
+ if (col == null) {
+ return null;
+ }
+
+ // Expressions takes a list of values as Objects
+ List<Object> values = new ArrayList<>();
+ for (int i = 1; i < predicate.getChildren().size(); ++i) {
+ if (!Expr.IS_LITERAL.apply(predicate.getChild(i))) {
+ return null;
+ }
+ LiteralExpr literal = (LiteralExpr) predicate.getChild(i);
+
+ // Cannot push IN or NOT_IN predicate with null literal values
+ if (Expr.IS_NULL_LITERAL.apply(literal)) {
+ return null;
+ }
+
+ Object value = getIcebergValue(ref, literal);
+ if (value == null) {
+ return null;
+ }
+
+ values.add(value);
+ }
+
+ // According to the method:
+ // 'org.apache.iceberg.expressions.InclusiveMetricsEvaluator.MetricsEvalVisitor#notIn'
+ // Expressions.notIn only works when the push-down column is the partition column
+ if (predicate.isNotIn())
+ return Expressions.notIn(col.getName(), values);
+ else {
+ return Expressions.in(col.getName(), values);
+ }
+ }
+
+ private UnboundPredicate<Object> convertIcebergPredicate(IsNullPredicate predicate) {
+ // Do not convert if there is an implicit cast
+ if (!(predicate.getChild(0) instanceof SlotRef)) {
+ return null;
+ }
+ SlotRef ref = (SlotRef) predicate.getChild(0);
+
+ // If predicate contains map/struct, this column would be null
+ Column col = ref.getDesc().getColumn();
+ if (col == null) {
+ return null;
+ }
+
+ if (predicate.isNotNull()) {
+ return Expressions.notNull(col.getName());
+ } else{
+ return Expressions.isNull(col.getName());
+ }
+ }
+
+ private Expression convertIcebergPredicate(CompoundPredicate predicate)
+ throws ImpalaException {
+ Operation op = getIcebergOperator(predicate.getOp());
+ if (op == null) {
+ return null;
+ }
+
+ Expression left = convertIcebergPredicate(predicate.getChild(0));
+ if (left == null) {
+ return null;
+ }
+ if (op.equals(Operation.NOT)) {
+ return Expressions.not(left);
+ }
+
+ Expression right = convertIcebergPredicate(predicate.getChild(1));
+ if (right == null) {
+ return null;
+ }
+ return op.equals(Operation.AND) ? Expressions.and(left, right)
+ : Expressions.or(left, right);
+ }
+
+ private Object getIcebergValue(SlotRef ref, LiteralExpr literal)
+ throws ImpalaException {
+ IcebergColumn iceCol = (IcebergColumn) ref.getDesc().getColumn();
+ Schema iceSchema = getIceTable().getIcebergSchema();
+ switch (literal.getType().getPrimitiveType()) {
+ case BOOLEAN: return ((BoolLiteral) literal).getValue();
+ case TINYINT:
+ case SMALLINT:
+ case INT: return ((NumericLiteral) literal).getIntValue();
+ case BIGINT: return ((NumericLiteral) literal).getLongValue();
+ case FLOAT: return (float) ((NumericLiteral) literal).getDoubleValue();
+ case DOUBLE: return ((NumericLiteral) literal).getDoubleValue();
+ case STRING:
+ case DATETIME:
+ case CHAR: return ((StringLiteral) literal).getUnescapedValue();
+ case TIMESTAMP: return getIcebergTsValue(literal, iceCol, iceSchema);
+ case DATE: return ((DateLiteral) literal).getValue();
+ case DECIMAL: return getIcebergDecimalValue(ref, (NumericLiteral) literal);
+ default: {
+ Preconditions.checkState(false,
+ "Unsupported iceberg type considered for predicate: %s",
+ literal.getType().toSql());
+ }
+ }
+ return null;
+ }
+
+ private BigDecimal getIcebergDecimalValue(SlotRef ref, NumericLiteral literal) {
+ Type colType = ref.getDesc().getColumn().getType();
+ int scale = colType.getDecimalDigits();
+ BigDecimal literalValue = literal.getValue();
+
+ if (literalValue.scale() > scale) return null;
+ // Iceberg DecimalLiteral needs to have the exact same scale.
+ if (literalValue.scale() < scale) return literalValue.setScale(scale);
+ return literalValue;
+ }
+
+ private Object getIcebergTsValue(LiteralExpr literal,
+ IcebergColumn iceCol, Schema iceSchema) throws AnalysisException {
+ try {
+ org.apache.iceberg.types.Type iceType = iceSchema.findType(iceCol.getFieldId());
+ Preconditions.checkState(iceType instanceof Types.TimestampType);
+ Types.TimestampType tsType = (Types.TimestampType) iceType;
+ if (tsType.shouldAdjustToUTC()) {
+ return ExprUtil.localTimestampToUnixTimeMicros(analyzer_, literal);
+ } else {
+ return ExprUtil.utcTimestampToUnixTimeMicros(analyzer_, literal);
+ }
+ } catch (InternalException ex) {
+ // We cannot interpret the timestamp literal. Maybe the timestamp is invalid,
+ // or the local timestamp ambigously converts to UTC due to daylight saving
+ // time backward turn. E.g. '2021-10-31 02:15:00 Europe/Budapest' converts to
+ // either '2021-10-31 00:15:00 UTC' or '2021-10-31 01:15:00 UTC'.
+ LOG.warn("Exception occurred during timestamp conversion: " + ex.toString() +
+ "\nThis means timestamp predicate is not pushed to Iceberg, let Impala " +
+ "backend handle it.");
+ }
+ return null;
+ }
+
+}
diff --git a/fe/src/main/java/org/apache/impala/planner/JoinNode.java b/fe/src/main/java/org/apache/impala/planner/JoinNode.java
index 0f3d259b8..f31045235 100644
--- a/fe/src/main/java/org/apache/impala/planner/JoinNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/JoinNode.java
@@ -84,10 +84,10 @@ public abstract class JoinNode extends PlanNode {
protected JoinTableId joinTableId_ = JoinTableId.INVALID;
// True if this join is used to do the join between insert and delete delta files.
- protected boolean isAcidJoin_ = false;
+ protected boolean isDeleteRowsJoin_ = false;
- public void setIsAcidJoin() {
- isAcidJoin_ = true;
+ public void setIsDeleteRowsJoin() {
+ isDeleteRowsJoin_ = true;
displayName_ = "DELETE EVENTS " + displayName_;
}
diff --git a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
index 143ff4bce..65dcb518a 100644
--- a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
@@ -1712,7 +1712,7 @@ public class SingleNodePlanner {
* Adds a new slot ref with path 'rawPath' to its tuple descriptor. This is a no-op if
* the tuple descriptor already has a slot ref with the given raw path.
*/
- private static void addSlotRefToDesc(Analyzer analyzer, List<String> rawPath)
+ public static void addSlotRefToDesc(Analyzer analyzer, List<String> rawPath)
throws AnalysisException {
Path resolvedPath = null;
try {
@@ -1773,7 +1773,7 @@ public class SingleNodePlanner {
acidJoinConjuncts, /*otherJoinConjuncts=*/Collections.emptyList());
acidJoin.setId(ctx_.getNextNodeId());
acidJoin.init(analyzer);
- acidJoin.setIsAcidJoin();
+ acidJoin.setIsDeleteRowsJoin();
return acidJoin;
}
@@ -1871,13 +1871,9 @@ public class SingleNodePlanner {
FeTable table = tblRef.getTable();
if (table instanceof FeFsTable) {
if (table instanceof FeIcebergTable) {
- FeFsTable feFsTable = ((FeIcebergTable) tblRef.getDesc().getTable()).
- getFeFsTable();
- analyzer.materializeSlots(conjuncts);
- scanNode = new IcebergScanNode(ctx_.getNextNodeId(), tblRef.getDesc(), conjuncts,
- tblRef, feFsTable, aggInfo);
- scanNode.init(analyzer);
- return scanNode;
+ IcebergScanPlanner icebergPlanner = new IcebergScanPlanner(analyzer, ctx_, tblRef,
+ conjuncts, aggInfo);
+ return icebergPlanner.createIcebergScanPlan();
}
return createHdfsScanPlan(tblRef, aggInfo, conjuncts, analyzer);
} else if (table instanceof FeDataSourceTable) {
diff --git a/fe/src/main/java/org/apache/impala/planner/UnionNode.java b/fe/src/main/java/org/apache/impala/planner/UnionNode.java
index 906438c2d..b1dbe7212 100644
--- a/fe/src/main/java/org/apache/impala/planner/UnionNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/UnionNode.java
@@ -99,6 +99,8 @@ public class UnionNode extends PlanNode {
*/
public boolean isConstantUnion() { return resultExprLists_.isEmpty(); }
+ public int getFirstNonPassthroughChildIndex() { return firstMaterializedChildIdx_; }
+
/**
* Add a child tree plus its corresponding unresolved resultExprs.
*/
diff --git a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
index f0ec43185..b4b4a7bda 100644
--- a/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
@@ -40,7 +40,9 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import org.apache.iceberg.ContentFile;
import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.FileFormat;
import org.apache.iceberg.FileScanTask;
import org.apache.iceberg.PartitionField;
@@ -525,33 +527,40 @@ public class IcebergUtil {
}
/**
- * Returns a Pair, the first element is the DataFiles by file system table location and
- * iceberg predicates, the second element is whether or not a DeleteFile exists for the
- * DataFiles in the first element.
+ * Returns a list of Iceberg FileScanTask objects. These objects contain a data file
+ * to be scanned and the associated delete files need to be applied.
*/
- public static Pair<List<DataFile>, Boolean> getIcebergDataFiles(FeIcebergTable table,
+ public static CloseableIterable<FileScanTask> planFiles(FeIcebergTable table,
List<Expression> predicates, TimeTravelSpec timeTravelSpec)
throws TableLoadingException {
- if (table.snapshotId() == -1) return new Pair<>(Collections.emptyList(), false);
+ if (table.snapshotId() == -1) return CloseableIterable.empty();
TableScan scan = createScanAsOf(table, timeTravelSpec);
for (Expression predicate : predicates) {
scan = scan.filter(predicate);
}
- List<DataFile> dataFileList = new ArrayList<>();
- boolean hasDeleteFile = false;
- try (CloseableIterable<FileScanTask> fileScanTasks = scan.planFiles()) {
- for (FileScanTask task : fileScanTasks) {
- if (!task.deletes().isEmpty()) {
- hasDeleteFile = true;
- }
- dataFileList.add(task.file());
+ return scan.planFiles();
+ }
+
+ /**
+ * Returns lists of data and delete files in the Iceberg table.
+ */
+ public static Pair<List<DataFile>, Set<DeleteFile>> getIcebergFiles(
+ FeIcebergTable table, List<Expression> predicates, TimeTravelSpec timeTravelSpec)
+ throws TableLoadingException {
+ List<DataFile> dataFiles = new ArrayList<>();
+ Set<DeleteFile> deleteFiles = new HashSet<>();
+ try (CloseableIterable<FileScanTask> fileScanTasks = planFiles(
+ table, predicates, timeTravelSpec)) {
+ for (FileScanTask scanTask : fileScanTasks) {
+ dataFiles.add(scanTask.file());
+ deleteFiles.addAll(scanTask.deletes());
}
} catch (IOException e) {
- throw new TableLoadingException("Data file list collection failed.", e);
+ throw new TableLoadingException("Error during reading Iceberg manifest files.", e);
}
- return new Pair<>(dataFileList, hasDeleteFile);
+ return new Pair<>(dataFiles, deleteFiles);
}
private static TableScan createScanAsOf(FeIcebergTable table,
@@ -583,11 +592,11 @@ public class IcebergUtil {
}
/**
- * Use DataFile path to generate 128-bit Murmur3 hash as map key, cached in memory
+ * Use ContentFile path to generate 128-bit Murmur3 hash as map key, cached in memory
*/
- public static String getDataFilePathHash(DataFile dataFile) {
+ public static String getFilePathHash(ContentFile contentFile) {
Hasher hasher = Hashing.murmur3_128().newHasher();
- hasher.putUnencodedChars(dataFile.path().toString());
+ hasher.putUnencodedChars(contentFile.path().toString());
return hasher.hash().toString();
}
@@ -868,9 +877,10 @@ public class IcebergUtil {
* It creates a flatbuffer so it can be passed between machines and processes without
* further de/serialization.
*/
- public static FbFileMetadata createIcebergMetadata(FeIcebergTable feTbl, DataFile df) {
+ public static FbFileMetadata createIcebergMetadata(FeIcebergTable feTbl,
+ ContentFile cf) {
FlatBufferBuilder fbb = new FlatBufferBuilder(1);
- int iceOffset = createIcebergMetadata(feTbl, fbb, df);
+ int iceOffset = createIcebergMetadata(feTbl, fbb, cf);
fbb.finish(FbFileMetadata.createFbFileMetadata(fbb, iceOffset));
ByteBuffer bb = fbb.dataBuffer().slice();
ByteBuffer compressedBb = ByteBuffer.allocate(bb.capacity());
@@ -879,17 +889,17 @@ public class IcebergUtil {
}
private static int createIcebergMetadata(FeIcebergTable feTbl, FlatBufferBuilder fbb,
- DataFile df) {
+ ContentFile cf) {
int partKeysOffset = -1;
- PartitionSpec spec = feTbl.getIcebergApiTable().specs().get(df.specId());
+ PartitionSpec spec = feTbl.getIcebergApiTable().specs().get(cf.specId());
if (spec != null && !spec.fields().isEmpty()) {
- partKeysOffset = createPartitionKeys(feTbl, fbb, spec, df);
+ partKeysOffset = createPartitionKeys(feTbl, fbb, spec, cf);
}
FbIcebergMetadata.startFbIcebergMetadata(fbb);
byte fileFormat = -1;
- if (df.format() == FileFormat.PARQUET) fileFormat = FbIcebergDataFileFormat.PARQUET;
- else if (df.format() == FileFormat.ORC) fileFormat = FbIcebergDataFileFormat.ORC;
- else if (df.format() == FileFormat.AVRO) fileFormat = FbIcebergDataFileFormat.AVRO;
+ if (cf.format() == FileFormat.PARQUET) fileFormat = FbIcebergDataFileFormat.PARQUET;
+ else if (cf.format() == FileFormat.ORC) fileFormat = FbIcebergDataFileFormat.ORC;
+ else if (cf.format() == FileFormat.AVRO) fileFormat = FbIcebergDataFileFormat.AVRO;
if (fileFormat != -1) {
FbIcebergMetadata.addFileFormat(fbb, fileFormat);
}
@@ -900,23 +910,23 @@ public class IcebergUtil {
}
private static int createPartitionKeys(FeIcebergTable feTbl, FlatBufferBuilder fbb,
- PartitionSpec spec, DataFile df) {
- Preconditions.checkState(spec.fields().size() == df.partition().size());
+ PartitionSpec spec, ContentFile cf) {
+ Preconditions.checkState(spec.fields().size() == cf.partition().size());
int[] partitionKeyOffsets = new int[spec.fields().size()];
for (int i = 0; i < spec.fields().size(); ++i) {
partitionKeyOffsets[i] =
- createPartitionTransformValue(feTbl, fbb, spec, df, i);
+ createPartitionTransformValue(feTbl, fbb, spec, cf, i);
}
return FbIcebergMetadata.createPartitionKeysVector(fbb, partitionKeyOffsets);
}
private static int createPartitionTransformValue(FeIcebergTable feTbl,
- FlatBufferBuilder fbb, PartitionSpec spec, DataFile df, int fieldIndex) {
+ FlatBufferBuilder fbb, PartitionSpec spec, ContentFile cf, int fieldIndex) {
PartitionField field = spec.fields().get(fieldIndex);
Pair<Byte, Integer> transform = getFbTransform(spec.schema(), field);
int valueOffset = -1;
if (transform.first != FbIcebergTransformType.VOID) {
- Object partValue = df.partition().get(fieldIndex, Object.class);
+ Object partValue = cf.partition().get(fieldIndex, Object.class);
String partValueString;
if (partValue != null) {
partValueString = partValue.toString();
diff --git a/fe/src/test/java/org/apache/impala/planner/PlannerTest.java b/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
index 7a7e60ebe..fc5650900 100644
--- a/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
+++ b/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
@@ -1230,6 +1230,16 @@ public class PlannerTest extends PlannerTestBase {
PlannerTestOption.DO_NOT_VALIDATE_ROWCOUNT_ESTIMATION_FOR_PARTITIONS));
}
+ /**
+ * Check that Iceberg V2 table scans work as expected.
+ */
+ @Test
+ public void testIcebergV2TableScans() {
+ runPlannerTestFile("iceberg-v2-tables", "functional_parquet",
+ ImmutableSet.of(
+ PlannerTestOption.DO_NOT_VALIDATE_ROWCOUNT_ESTIMATION_FOR_PARTITIONS));
+ }
+
/**
* Test limit pushdown into analytic sort in isolation.
*/
diff --git a/testdata/data/README b/testdata/data/README
index 13ff1162c..21a5cd9b6 100644
--- a/testdata/data/README
+++ b/testdata/data/README
@@ -762,4 +762,107 @@ Generated and migrated by Hive
CREATE TABLE iceberg_migrated_complex_test_orc (struct_1_col struct<int_array_col: array<int>, string_col: string, bool_int_map_col: map<boolean, int>>, int_bigint_map_col map<int, bigint>, struct_2_col struct<struct_3_col: struct<float_col: float, string_double_map_col: map<string, double>, bigint_array_col: array<bigint>>, int_int_map_col: map<int, int>>) stored as orc;
insert into table iceberg_migrated_complex_test_orc values (named_struct("int_array_col", array(0), "string_col", "A", "bool_int_map_col", map(True, 1 )), map(2,CAST(3 as bigint)), named_struct("struct_3_col", named_struct("float_col", cast(0.5 as float), "string_double_map_col", map("B", cast(1.5 as double)), "bigint_array_col", array(cast(4 as bigint))), "int_int_map_col", map(5,6)));
ALTER TABLE iceberg_migrated_complex_test_orc SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler');
-Then extracted from hdfs and modified to be able to load as an external hadoop table
\ No newline at end of file
+Then extracted from hdfs and modified to be able to load as an external hadoop table
+
+iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes:
+Created by Hive 3.1.3000.2022.0.10.0-49 r5cd8759d0df2ecbfb788b7f4ee0edce6022ee459
+create table iceberg_v2_no_deletes (i int, s string)
+stored by iceberg
+location '/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes'
+tblproperties ('format-version'='2');
+insert into iceberg_v2_no_deletes values (1, 'x'), (2, 'y'), (3, 'z');
+(setting table location is important because it makes it easier the conversion later).
+Then saved the contents from HDFS to local ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice
+And converted the HiveCatalog metadata to HadoopCatalog metadata via the following scripts:
+convert_to_iceberg.sh:
+#!/bin/bash
+
+i=0
+for f in *.json; do
+ i=$((i+1))
+ sed -i 's|hdfs://localhost:20500/test-warehouse/|/test-warehouse/|' $f
+ mv $f v${i}.metadata.json
+done
+echo ${i} > version-hint.txt
+
+for f in *.avro; do
+ avro_iceberg_convert.sh $f 'hdfs://localhost:20500/test-warehouse/' '/test-warehouse/'
+ mv ${f}_mod $f
+done
+
+rm *.avro_json
+
+avro_iceberg_convert.sh:
+#!/bin/bash
+
+# Usage: avro_iceberg_convert.sh <source avro> <search string> <replace string>
+# Example:
+SOURCE_FILE=$1
+SEARCH_STR=$2
+REPLACE_STR=$3
+TMP_JSON=$1_json
+DST_AVRO=$1_mod
+AVRO_TOOLS="/path/to/avro-tools-1.11.0.jar"
+
+if [ ! -f "$AVRO_TOOLS" ]; then
+ echo "Can't find $AVRO_TOOLS."
+ exit
+fi
+if [ ! -f $SOURCE_FILE ]; then
+ echo "Can't find source file: $SOURCE_FILE!"
+ exit
+fi
+# Transform avro to json:
+java -jar $AVRO_TOOLS tojson --pretty $SOURCE_FILE > $TMP_JSON
+# Replace search string with replace string
+sed --in-place "s|$SEARCH_STR|$REPLACE_STR|g" $TMP_JSON
+# Convert the file back to avro
+SCHEMA=`java -jar $AVRO_TOOLS getschema $SOURCE_FILE`
+java -jar $AVRO_TOOLS fromjson $TMP_JSON --schema "$SCHEMA" > $DST_AVRO
+
+iceberg_v2_partitioned_position_deletes:
+iceberg_v2_partitioned_position_deletes_orc:
+Created similarly to iceberg_v2_no_deletes.
+Hive> create table iceberg_v2_partitioned_position_deletes (
+ id int, `user` string, action string, event_time timestamp)
+partitioned by spec (action)
+STORED BY ICEBERG
+location '/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes'
+tblproperties ('format-version'='2', 'write.format.default'='parquet');
+Impala> insert into iceberg_v2_partitioned_position_deletes select * from functional_parquet.iceberg_partitioned;
+Hive> delete from iceberg_v2_partitioned_position_deletes where id % 2 = 1;
+ORC table is similarly created.
+
+
+iceberg_v2_positional_delete_all_rows:
+iceberg_v2_positional_delete_all_rows_orc:
+Created similarly to iceberg_v2_no_deletes.
+Hive> create table iceberg_v2_positional_delete_all_rows_orc (
+ i int, s string)
+ stored by iceberg
+ location '/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc'
+ tblproperties ('format-version'='2', 'write.format.default'='orc');
+Hive> insert into iceberg_v2_positional_delete_all_rows_orc values (1, 'x'), (2, 'y'), (3, 'z');
+Hive> delete from iceberg_v2_positional_delete_all_rows_orc;
+
+iceberg_v2_positional_not_all_data_files_have_delete_files:
+iceberg_v2_positional_not_all_data_files_have_delete_files_orc:
+Created similarly to iceberg_v2_no_deletes.
+create table iceberg_v2_positional_not_all_data_files_have_delete_files_orc (i int, s string)
+stored by iceberg
+location '/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc'
+tblproperties ('format-version'='2', 'write.format.default'='orc');
+insert into iceberg_v2_positional_not_all_data_files_have_delete_files_orc values (1,'a'), (2,'b'), (3,'c');
+insert into iceberg_v2_positional_not_all_data_files_have_delete_files_orc values (4,'d'), (5,'e'), (6,'f');
+insert into iceberg_v2_positional_not_all_data_files_have_delete_files_orc values (7,'g'), (8,'h'), (9,'i');
+update iceberg_v2_positional_not_all_data_files_have_delete_files_orc set s='X' where i = 5;
+delete from iceberg_v2_positional_not_all_data_files_have_delete_files_orc where i > 6;
+
+iceberg_v2_positional_update_all_rows:
+Created similarly to iceberg_v2_no_deletes
+create table iceberg_v2_positional_update_all_rows (i int, s string)
+stored by iceberg
+location '/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows'
+tblproperties ('format-version'='2');
+insert into iceberg_v2_positional_update_all_rows values (1,'a'), (2,'b'), (3,'c')
+update iceberg_v2_positional_update_all_rows set s = upper(s);
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet
index ecd146b49..2fb34f0b4 100644
Binary files a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/data/00000-0-data-boroknagyz_20220819180420_a7e5a731-8762-4b59-b3f2-fe6f065cf59b-job_16597105613620_0031-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/data/00000-0-data-boroknagyz_20220819180420_a7e5a731-8762-4b59-b3f2-fe6f065cf59b-job_16597105613620_0031-00001.parquet
new file mode 100644
index 000000000..75a9778ff
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/data/00000-0-data-boroknagyz_20220819180420_a7e5a731-8762-4b59-b3f2-fe6f065cf59b-job_16597105613620_0031-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro
new file mode 100644
index 000000000..75e07a796
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/5c80922f-01b5-4d52-bc93-6505be3b977b-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro
new file mode 100644
index 000000000..412c9b6bb
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v1.metadata.json
new file mode 100644
index 000000000..557f3e8fa
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v1.metadata.json
@@ -0,0 +1,45 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "5320f4a3-e183-407b-a894-78e4b91f9dce",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1660924995601,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v2.metadata.json
new file mode 100644
index 000000000..1e95b6d79
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/v2.metadata.json
@@ -0,0 +1,70 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "5320f4a3-e183-407b-a894-78e4b91f9dce",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1660925066239,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 728158873687794725,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 728158873687794725,
+ "timestamp-ms" : 1660925066239,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/snap-728158873687794725-1-5c80922f-01b5-4d52-bc93-6505be3b977b.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660925066239,
+ "snapshot-id" : 728158873687794725
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660924995601,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/00000-cfa781e2-dcd3-4555-b603-78a4ea08f3bc.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/version-hint.txt
new file mode 100644
index 000000000..0cfbf0888
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes/metadata/version-hint.txt
@@ -0,0 +1 @@
+2
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/data/00000-0-data-boroknagyz_20220830153046_7c9aaa51-d601-4c5b-a51c-67c1e9e1fe29-job_16618645111570_0001-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/data/00000-0-data-boroknagyz_20220830153046_7c9aaa51-d601-4c5b-a51c-67c1e9e1fe29-job_16618645111570_0001-00001.orc
new file mode 100644
index 000000000..41975d3f9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/data/00000-0-data-boroknagyz_20220830153046_7c9aaa51-d601-4c5b-a51c-67c1e9e1fe29-job_16618645111570_0001-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/a72290c9-c518-4719-8502-6c83a881de07-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/a72290c9-c518-4719-8502-6c83a881de07-m0.avro
new file mode 100644
index 000000000..ebda019ff
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/a72290c9-c518-4719-8502-6c83a881de07-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/snap-1041485290740594175-1-a72290c9-c518-4719-8502-6c83a881de07.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/snap-1041485290740594175-1-a72290c9-c518-4719-8502-6c83a881de07.avro
new file mode 100644
index 000000000..d0b9875f0
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/snap-1041485290740594175-1-a72290c9-c518-4719-8502-6c83a881de07.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v1.metadata.json
new file mode 100644
index 000000000..799366861
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v1.metadata.json
@@ -0,0 +1,46 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "1c09f477-4a6d-4ca0-ae49-a217f7ee89ff",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1661866210693,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "write.format.default" : "orc",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v2.metadata.json
new file mode 100644
index 000000000..e58c4d2fc
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/v2.metadata.json
@@ -0,0 +1,71 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "1c09f477-4a6d-4ca0-ae49-a217f7ee89ff",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1661866251868,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 1041485290740594175,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 1041485290740594175,
+ "timestamp-ms" : 1661866251868,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/snap-1041485290740594175-1-a72290c9-c518-4719-8502-6c83a881de07.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661866251868,
+ "snapshot-id" : 1041485290740594175
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661866210693,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/00000-d129a4cf-31c5-4397-abb3-26073e00881b.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/version-hint.txt
new file mode 100644
index 000000000..0cfbf0888
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc/metadata/version-hint.txt
@@ -0,0 +1 @@
+2
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00002.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00002.parquet
new file mode 100644
index 000000000..17dcea783
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00002.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/874b32d9a15da206-f60e01cb00000003_1034098606_data.0.parq b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/874b32d9a15da206-f60e01cb00000003_1034098606_data.0.parq
new file mode 100644
index 000000000..3a963e1ff
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=click/874b32d9a15da206-f60e01cb00000003_1034098606_data.0.parq differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00003.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00003.parquet
new file mode 100644
index 000000000..21da1678e
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00003.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/874b32d9a15da206-f60e01cb00000003_1489587766_data.0.parq b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/874b32d9a15da206-f60e01cb00000003_1489587766_data.0.parq
new file mode 100644
index 000000000..af582b01c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=download/874b32d9a15da206-f60e01cb00000003_1489587766_data.0.parq differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00001.parquet
new file mode 100644
index 000000000..4ddf1b597
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/00000-0-delete-boroknagyz_20220819183231_cfc565f5-52b9-4669-9f69-d29c50a84a5e-job_16597105613621_0032-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/874b32d9a15da206-f60e01cb00000004_1711435901_data.0.parq b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/874b32d9a15da206-f60e01cb00000004_1711435901_data.0.parq
new file mode 100644
index 000000000..63866af23
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/data/action=view/874b32d9a15da206-f60e01cb00000004_1711435901_data.0.parq differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro
new file mode 100644
index 000000000..8a74282d9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/464c179e-c9ba-40f5-a35f-144106a1f16c-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro
new file mode 100644
index 000000000..6fbb84a02
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/771485e9-78ac-4ffc-b1ef-1fda5bab33cf-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro
new file mode 100644
index 000000000..8cdfd958c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-8885697082976537578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-8885697082976537578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro
new file mode 100644
index 000000000..4e5f12109
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-8885697082976537578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v1.metadata.json
new file mode 100644
index 000000000..7ded28e80
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v1.metadata.json
@@ -0,0 +1,61 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "f575f129-24aa-41d0-ba95-029fd5aad53a",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1660926675125,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "external.table.purge" : "TRUE",
+ "write.format.default" : "parquet",
+ "OBJCAPABILITIES" : "EXTREAD,EXTWRITE",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v2.metadata.json
new file mode 100644
index 000000000..8f52680d6
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v2.metadata.json
@@ -0,0 +1,88 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "f575f129-24aa-41d0-ba95-029fd5aad53a",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1660926726755,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "impala.events.catalogServiceId" : "8fb97d22f0444a1a:b53634aa96205845",
+ "external.table.purge" : "TRUE",
+ "impala.events.catalogVersion" : "17938",
+ "write.format.default" : "parquet",
+ "OBJCAPABILITIES" : "EXTREAD,EXTWRITE",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 2057976186205897384,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 2057976186205897384,
+ "timestamp-ms" : 1660926726752,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "3",
+ "added-records" : "20",
+ "added-files-size" : "3568",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "3568",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660926726752,
+ "snapshot-id" : 2057976186205897384
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660926675125,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/00000-cea1563c-d3c3-4b75-98ed-2537574012a8.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v3.metadata.json
new file mode 100644
index 000000000..fc59d7364
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/v3.metadata.json
@@ -0,0 +1,114 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "f575f129-24aa-41d0-ba95-029fd5aad53a",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1660926760575,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "impala.events.catalogServiceId" : "8fb97d22f0444a1a:b53634aa96205845",
+ "external.table.purge" : "TRUE",
+ "impala.events.catalogVersion" : "17938",
+ "write.format.default" : "parquet",
+ "OBJCAPABILITIES" : "EXTREAD,EXTWRITE",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 8885697082976537578,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 2057976186205897384,
+ "timestamp-ms" : 1660926726752,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "3",
+ "added-records" : "20",
+ "added-files-size" : "3568",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "3568",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-2057976186205897384-1-771485e9-78ac-4ffc-b1ef-1fda5bab33cf.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8885697082976537578,
+ "parent-snapshot-id" : 2057976186205897384,
+ "timestamp-ms" : 1660926760575,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "3",
+ "added-files-size" : "9695",
+ "added-position-deletes" : "10",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "13263",
+ "total-data-files" : "3",
+ "total-delete-files" : "3",
+ "total-position-deletes" : "10",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/snap-8885697082976537578-1-464c179e-c9ba-40f5-a35f-144106a1f16c.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660926726752,
+ "snapshot-id" : 2057976186205897384
+ }, {
+ "timestamp-ms" : 1660926760575,
+ "snapshot-id" : 8885697082976537578
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660926675125,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/00000-cea1563c-d3c3-4b75-98ed-2537574012a8.metadata.json"
+ }, {
+ "timestamp-ms" : 1660926726755,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/00001-dbc31643-8820-4b7b-ab50-2860353bea15.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/version-hint.txt
new file mode 100644
index 000000000..00750edc0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes/metadata/version-hint.txt
@@ -0,0 +1 @@
+3
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00001.orc
new file mode 100644
index 000000000..90acd03b8
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00002.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00002.orc
new file mode 100644
index 000000000..eddc67562
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=click/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00002.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00002.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00002.orc
new file mode 100644
index 000000000..93f68fda6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00002.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00003.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00003.orc
new file mode 100644
index 000000000..46cf4b2cd
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=download/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00003.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00003.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00003.orc
new file mode 100644
index 000000000..d0176338e
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-data-boroknagyz_20220830173852_560d6f0e-8508-42d6-95a7-67bfa860a28c-job_16618645111571_0006-00003.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00001.orc
new file mode 100644
index 000000000..be9fe87b8
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/data/action=view/00000-0-delete-boroknagyz_20220830173925_010bf879-cf2f-4209-a70b-1a8309657106-job_16618645111571_0006-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro
new file mode 100644
index 000000000..621522185
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/588ee1ca-6a85-4af2-8ba2-e595e71712ba-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro
new file mode 100644
index 000000000..db09bdbdf
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/5b14eaa8-83cb-4f71-a473-402f345fa5b5-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5359840930115020310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5359840930115020310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro
new file mode 100644
index 000000000..82096ec3c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5359840930115020310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro
new file mode 100644
index 000000000..fbb9e1bde
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v1.metadata.json
new file mode 100644
index 000000000..e05b020e3
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v1.metadata.json
@@ -0,0 +1,61 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "72375e0f-f012-4f23-ab77-ee114689e86b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1661873778765,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "write.format.default" : "orc",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v2.metadata.json
new file mode 100644
index 000000000..09dd3622d
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v2.metadata.json
@@ -0,0 +1,86 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "72375e0f-f012-4f23-ab77-ee114689e86b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1661873942863,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 5416468273053855108,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5416468273053855108,
+ "timestamp-ms" : 1661873942863,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "3",
+ "added-records" : "20",
+ "added-files-size" : "1807",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "1807",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661873942863,
+ "snapshot-id" : 5416468273053855108
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661873778765,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/00000-79bb9f0f-a28c-431f-9d7b-6afb8362cb1e.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v3.metadata.json
new file mode 100644
index 000000000..a3f8d6cd7
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/v3.metadata.json
@@ -0,0 +1,112 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "72375e0f-f012-4f23-ab77-ee114689e86b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1661873971614,
+ "last-column-id" : 4,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "id",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "user",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 3,
+ "name" : "action",
+ "required" : false,
+ "type" : "string"
+ }, {
+ "id" : 4,
+ "name" : "event_time",
+ "required" : false,
+ "type" : "timestamp"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ {
+ "name" : "action",
+ "transform" : "identity",
+ "source-id" : 3,
+ "field-id" : 1000
+ } ]
+ } ],
+ "last-partition-id" : 1000,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 5359840930115020310,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5416468273053855108,
+ "timestamp-ms" : 1661873942863,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "3",
+ "added-records" : "20",
+ "added-files-size" : "1807",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "1807",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5416468273053855108-1-588ee1ca-6a85-4af2-8ba2-e595e71712ba.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5359840930115020310,
+ "parent-snapshot-id" : 5416468273053855108,
+ "timestamp-ms" : 1661873971614,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "3",
+ "added-files-size" : "4882",
+ "added-position-deletes" : "10",
+ "changed-partition-count" : "3",
+ "total-records" : "20",
+ "total-files-size" : "6689",
+ "total-data-files" : "3",
+ "total-delete-files" : "3",
+ "total-position-deletes" : "10",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/snap-5359840930115020310-1-5b14eaa8-83cb-4f71-a473-402f345fa5b5.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661873942863,
+ "snapshot-id" : 5416468273053855108
+ }, {
+ "timestamp-ms" : 1661873971614,
+ "snapshot-id" : 5359840930115020310
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661873778765,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/00000-79bb9f0f-a28c-431f-9d7b-6afb8362cb1e.metadata.json"
+ }, {
+ "timestamp-ms" : 1661873942863,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/00001-92cfbe35-7e46-4e99-bd44-544c063cef9e.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/version-hint.txt
new file mode 100644
index 000000000..00750edc0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc/metadata/version-hint.txt
@@ -0,0 +1 @@
+3
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-data-boroknagyz_20220819150927_e95cefe8-5549-42f6-98a6-b1bbea9a4224-job_16597105613620_0025-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-data-boroknagyz_20220819150927_e95cefe8-5549-42f6-98a6-b1bbea9a4224-job_16597105613620_0025-00001.parquet
new file mode 100644
index 000000000..17002437c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-data-boroknagyz_20220819150927_e95cefe8-5549-42f6-98a6-b1bbea9a4224-job_16597105613620_0025-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-delete-boroknagyz_20220819150939_8f0789df-fcdb-4ba8-875c-4aeac53f30b9-job_16597105613621_0025-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-delete-boroknagyz_20220819150939_8f0789df-fcdb-4ba8-875c-4aeac53f30b9-job_16597105613621_0025-00001.parquet
new file mode 100644
index 000000000..eeffdcbe7
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/data/00000-0-delete-boroknagyz_20220819150939_8f0789df-fcdb-4ba8-875c-4aeac53f30b9-job_16597105613621_0025-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro
new file mode 100644
index 000000000..6af2c8cd6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/236523f7-a5bc-459f-b4c9-16af5bd43bca-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro
new file mode 100644
index 000000000..7db60b917
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/816400dd-012d-40c5-ab65-bc16ff18d2d7-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-444149380144800647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-444149380144800647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro
new file mode 100644
index 000000000..2076f1b7c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-444149380144800647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro
new file mode 100644
index 000000000..e31a4d860
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v1.metadata.json
new file mode 100644
index 000000000..408e19c42
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v1.metadata.json
@@ -0,0 +1,45 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "574fa22d-ffd7-4ee5-984f-5a4b2d1d5c84",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1660914560473,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v2.metadata.json
new file mode 100644
index 000000000..070337ccb
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v2.metadata.json
@@ -0,0 +1,70 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "574fa22d-ffd7-4ee5-984f-5a4b2d1d5c84",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1660914572575,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 8593920101374128463,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 8593920101374128463,
+ "timestamp-ms" : 1660914572575,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660914572575,
+ "snapshot-id" : 8593920101374128463
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660914560473,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/00000-2bb68bdc-ac69-4fc7-9647-a09f3ee32c4f.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v3.metadata.json
new file mode 100644
index 000000000..b78da3d22
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/v3.metadata.json
@@ -0,0 +1,96 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "574fa22d-ffd7-4ee5-984f-5a4b2d1d5c84",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1660914584689,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 444149380144800647,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 8593920101374128463,
+ "timestamp-ms" : 1660914572575,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-8593920101374128463-1-236523f7-a5bc-459f-b4c9-16af5bd43bca.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 444149380144800647,
+ "parent-snapshot-id" : 8593920101374128463,
+ "timestamp-ms" : 1660914584689,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "2666",
+ "added-position-deletes" : "3",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "3291",
+ "total-data-files" : "1",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "3",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/snap-444149380144800647-1-816400dd-012d-40c5-ab65-bc16ff18d2d7.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660914572575,
+ "snapshot-id" : 8593920101374128463
+ }, {
+ "timestamp-ms" : 1660914584689,
+ "snapshot-id" : 444149380144800647
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660914560473,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/00000-2bb68bdc-ac69-4fc7-9647-a09f3ee32c4f.metadata.json"
+ }, {
+ "timestamp-ms" : 1660914572575,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/00001-401d81b6-63dc-4df8-8f19-ef30f48d0d4e.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/version-hint.txt
new file mode 100644
index 000000000..00750edc0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows/metadata/version-hint.txt
@@ -0,0 +1 @@
+3
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-data-boroknagyz_20220830153712_a4060925-000f-4454-b68b-8f7570d3aa7c-job_16618645111570_0002-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-data-boroknagyz_20220830153712_a4060925-000f-4454-b68b-8f7570d3aa7c-job_16618645111570_0002-00001.orc
new file mode 100644
index 000000000..41975d3f9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-data-boroknagyz_20220830153712_a4060925-000f-4454-b68b-8f7570d3aa7c-job_16618645111570_0002-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-delete-boroknagyz_20220830153733_8f03a46b-e4a5-4643-896b-a1cc5b09cf2b-job_16618645111571_0002-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-delete-boroknagyz_20220830153733_8f03a46b-e4a5-4643-896b-a1cc5b09cf2b-job_16618645111571_0002-00001.orc
new file mode 100644
index 000000000..36d2b72f1
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/data/00000-0-delete-boroknagyz_20220830153733_8f03a46b-e4a5-4643-896b-a1cc5b09cf2b-job_16618645111571_0002-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro
new file mode 100644
index 000000000..6dca3a00c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro
new file mode 100644
index 000000000..a9d0b545d
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/83d17f01-336e-41ab-a791-ffd5f511f6ab-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-1801547319505512253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-1801547319505512253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro
new file mode 100644
index 000000000..898df792e
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-1801547319505512253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro
new file mode 100644
index 000000000..6c7641117
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v1.metadata.json
new file mode 100644
index 000000000..31478737c
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v1.metadata.json
@@ -0,0 +1,46 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "c1985299-1304-4b78-8860-040a2c6dc4ff",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1661866573782,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "write.format.default" : "orc",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v2.metadata.json
new file mode 100644
index 000000000..d08efaecc
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v2.metadata.json
@@ -0,0 +1,71 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "c1985299-1304-4b78-8860-040a2c6dc4ff",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1661866640281,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 4807054508647143162,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4807054508647143162,
+ "timestamp-ms" : 1661866640281,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661866640281,
+ "snapshot-id" : 4807054508647143162
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661866573782,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/00000-49f36a40-e2b6-411f-9efe-0519b28b8774.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v3.metadata.json
new file mode 100644
index 000000000..6d53f883c
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/v3.metadata.json
@@ -0,0 +1,97 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "c1985299-1304-4b78-8860-040a2c6dc4ff",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1661866659401,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 1801547319505512253,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4807054508647143162,
+ "timestamp-ms" : 1661866640281,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-4807054508647143162-1-83d17f01-336e-41ab-a791-ffd5f511f6ab.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 1801547319505512253,
+ "parent-snapshot-id" : 4807054508647143162,
+ "timestamp-ms" : 1661866659401,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "1319",
+ "added-position-deletes" : "3",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "1659",
+ "total-data-files" : "1",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "3",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/snap-1801547319505512253-1-6d54c2e9-a4c7-4c5c-8a6f-a17f92de4c48.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661866640281,
+ "snapshot-id" : 4807054508647143162
+ }, {
+ "timestamp-ms" : 1661866659401,
+ "snapshot-id" : 1801547319505512253
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661866573782,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/00000-49f36a40-e2b6-411f-9efe-0519b28b8774.metadata.json"
+ }, {
+ "timestamp-ms" : 1661866640281,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/00001-e3771417-22a9-42dd-ac01-a2e64826dc12.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/version-hint.txt
new file mode 100644
index 000000000..00750edc0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc/metadata/version-hint.txt
@@ -0,0 +1 @@
+3
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154646_1cad8c38-c65e-4c7c-b516-c4d9faf82448-job_16597105613620_0026-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154646_1cad8c38-c65e-4c7c-b516-c4d9faf82448-job_16597105613620_0026-00001.parquet
new file mode 100644
index 000000000..ca6f0c4f5
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154646_1cad8c38-c65e-4c7c-b516-c4d9faf82448-job_16597105613620_0026-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154718_db95aeae-d530-4fba-8336-e47fa712b987-job_16597105613620_0026-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154718_db95aeae-d530-4fba-8336-e47fa712b987-job_16597105613620_0026-00001.parquet
new file mode 100644
index 000000000..37ff982ee
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154718_db95aeae-d530-4fba-8336-e47fa712b987-job_16597105613620_0026-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154733_162cd69d-80e2-425b-9638-9008d0937573-job_16597105613620_0026-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154733_162cd69d-80e2-425b-9638-9008d0937573-job_16597105613620_0026-00001.parquet
new file mode 100644
index 000000000..5c70be4ff
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154733_162cd69d-80e2-425b-9638-9008d0937573-job_16597105613620_0026-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613620_0026-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613620_0026-00001.parquet
new file mode 100644
index 000000000..0a096a801
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-data-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613620_0026-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613621_0026-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613621_0026-00001.parquet
new file mode 100644
index 000000000..5ae787509
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819154922_62429d29-6c44-4707-b348-ac189b8d79d3-job_16597105613621_0026-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819155000_b0411d64-66e3-49fd-a2f0-dab69282a896-job_16597105613621_0027-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819155000_b0411d64-66e3-49fd-a2f0-dab69282a896-job_16597105613621_0027-00001.parquet
new file mode 100644
index 000000000..a533e9483
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/data/00000-0-delete-boroknagyz_20220819155000_b0411d64-66e3-49fd-a2f0-dab69282a896-job_16597105613621_0027-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro
new file mode 100644
index 000000000..172d5a278
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/969de65c-8915-4ae5-8d54-a82701195c55-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/969de65c-8915-4ae5-8d54-a82701195c55-m0.avro
new file mode 100644
index 000000000..6c1732a5f
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/969de65c-8915-4ae5-8d54-a82701195c55-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro
new file mode 100644
index 000000000..73ea21853
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/c4e37595-32c2-483f-b6d7-866f60e36976-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro
new file mode 100644
index 000000000..29aaccdeb
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/db6f17fe-6fb6-4120-839d-4d6ca5244a1c-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro
new file mode 100644
index 000000000..6a1762f5f
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ea749da1-7b98-4dca-a4eb-f7d5d62f9dde-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro
new file mode 100644
index 000000000..cedae8137
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-1497619269847778439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-1497619269847778439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro
new file mode 100644
index 000000000..48a5703c9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-1497619269847778439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro
new file mode 100644
index 000000000..283b9c0e3
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro
new file mode 100644
index 000000000..2e4dcb07d
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro
new file mode 100644
index 000000000..00e026b13
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro
new file mode 100644
index 000000000..c3a2cccea
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro
new file mode 100644
index 000000000..ced9a6bed
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v1.metadata.json
new file mode 100644
index 000000000..6838c4c9d
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v1.metadata.json
@@ -0,0 +1,45 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1660916767386,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v2.metadata.json
new file mode 100644
index 000000000..adf852360
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v2.metadata.json
@@ -0,0 +1,70 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1660916807090,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 4363979609026842966,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v3.metadata.json
new file mode 100644
index 000000000..402b29482
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v3.metadata.json
@@ -0,0 +1,96 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1660916844291,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 5762682948883272650,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5762682948883272650,
+ "parent-snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916844291,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "snapshot-id" : 5762682948883272650
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916807090,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00001-3830c2b9-94ba-443f-8c17-aa9bfb9106f8.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v4.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v4.metadata.json
new file mode 100644
index 000000000..9a4b9b4bb
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v4.metadata.json
@@ -0,0 +1,122 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 3,
+ "last-updated-ms" : 1660916858288,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 7508485421322116327,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5762682948883272650,
+ "parent-snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916844291,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 7508485421322116327,
+ "parent-snapshot-id" : 5762682948883272650,
+ "timestamp-ms" : 1660916858288,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "9",
+ "total-files-size" : "1875",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "snapshot-id" : 5762682948883272650
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "snapshot-id" : 7508485421322116327
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916807090,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00001-3830c2b9-94ba-443f-8c17-aa9bfb9106f8.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00002-3d329e58-1eb2-465a-ba51-702884163d6a.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v5.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v5.metadata.json
new file mode 100644
index 000000000..3db3f8dad
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v5.metadata.json
@@ -0,0 +1,148 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 4,
+ "last-updated-ms" : 1660916969175,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 7490459762454857930,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5762682948883272650,
+ "parent-snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916844291,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 7508485421322116327,
+ "parent-snapshot-id" : 5762682948883272650,
+ "timestamp-ms" : 1660916858288,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "9",
+ "total-files-size" : "1875",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 7490459762454857930,
+ "parent-snapshot-id" : 7508485421322116327,
+ "timestamp-ms" : 1660916969175,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "620",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "2495",
+ "total-data-files" : "4",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "snapshot-id" : 5762682948883272650
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "snapshot-id" : 7508485421322116327
+ }, {
+ "timestamp-ms" : 1660916969175,
+ "snapshot-id" : 7490459762454857930
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916807090,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00001-3830c2b9-94ba-443f-8c17-aa9bfb9106f8.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00002-3d329e58-1eb2-465a-ba51-702884163d6a.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00003-c2da6c8f-addf-4b50-8e24-f195803dc7b7.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v6.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v6.metadata.json
new file mode 100644
index 000000000..c72f735f9
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v6.metadata.json
@@ -0,0 +1,174 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 5,
+ "last-updated-ms" : 1660916969420,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 752781918366351945,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5762682948883272650,
+ "parent-snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916844291,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 7508485421322116327,
+ "parent-snapshot-id" : 5762682948883272650,
+ "timestamp-ms" : 1660916858288,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "9",
+ "total-files-size" : "1875",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 7490459762454857930,
+ "parent-snapshot-id" : 7508485421322116327,
+ "timestamp-ms" : 1660916969175,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "620",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "2495",
+ "total-data-files" : "4",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 5,
+ "snapshot-id" : 752781918366351945,
+ "parent-snapshot-id" : 7490459762454857930,
+ "timestamp-ms" : 1660916969420,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "2697",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "5192",
+ "total-data-files" : "4",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "snapshot-id" : 5762682948883272650
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "snapshot-id" : 7508485421322116327
+ }, {
+ "timestamp-ms" : 1660916969175,
+ "snapshot-id" : 7490459762454857930
+ }, {
+ "timestamp-ms" : 1660916969420,
+ "snapshot-id" : 752781918366351945
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916807090,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00001-3830c2b9-94ba-443f-8c17-aa9bfb9106f8.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00002-3d329e58-1eb2-465a-ba51-702884163d6a.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00003-c2da6c8f-addf-4b50-8e24-f195803dc7b7.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916969175,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00004-9d984fc8-1528-4942-b93f-14146ba7007a.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v7.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v7.metadata.json
new file mode 100644
index 000000000..e93219bc0
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/v7.metadata.json
@@ -0,0 +1,200 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "843f856b-c95c-4297-87c4-1eccf9a2608e",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files",
+ "last-sequence-number" : 6,
+ "last-updated-ms" : 1660917010112,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 1497619269847778439,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916807090,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-4363979609026842966-1-db6f17fe-6fb6-4120-839d-4d6ca5244a1c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5762682948883272650,
+ "parent-snapshot-id" : 4363979609026842966,
+ "timestamp-ms" : 1660916844291,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-5762682948883272650-1-969de65c-8915-4ae5-8d54-a82701195c55.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 7508485421322116327,
+ "parent-snapshot-id" : 5762682948883272650,
+ "timestamp-ms" : 1660916858288,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "9",
+ "total-files-size" : "1875",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7508485421322116327-1-4ba5ef32-7f8b-4418-b5bf-3fb8002e0dde.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 7490459762454857930,
+ "parent-snapshot-id" : 7508485421322116327,
+ "timestamp-ms" : 1660916969175,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "620",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "2495",
+ "total-data-files" : "4",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-7490459762454857930-1-ec9e7ecc-b546-42dd-8d0d-0dde2182dbc7.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 5,
+ "snapshot-id" : 752781918366351945,
+ "parent-snapshot-id" : 7490459762454857930,
+ "timestamp-ms" : 1660916969420,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "2697",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "5192",
+ "total-data-files" : "4",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-752781918366351945-2-c4e37595-32c2-483f-b6d7-866f60e36976.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 6,
+ "snapshot-id" : 1497619269847778439,
+ "parent-snapshot-id" : 752781918366351945,
+ "timestamp-ms" : 1660917010112,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "2760",
+ "added-position-deletes" : "3",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "7952",
+ "total-data-files" : "4",
+ "total-delete-files" : "2",
+ "total-position-deletes" : "4",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/snap-1497619269847778439-1-ea749da1-7b98-4dca-a4eb-f7d5d62f9dde.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660916807090,
+ "snapshot-id" : 4363979609026842966
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "snapshot-id" : 5762682948883272650
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "snapshot-id" : 7508485421322116327
+ }, {
+ "timestamp-ms" : 1660916969175,
+ "snapshot-id" : 7490459762454857930
+ }, {
+ "timestamp-ms" : 1660916969420,
+ "snapshot-id" : 752781918366351945
+ }, {
+ "timestamp-ms" : 1660917010112,
+ "snapshot-id" : 1497619269847778439
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660916767386,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00000-6230f74b-1ded-47a2-a037-991757bb7dd9.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916807090,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00001-3830c2b9-94ba-443f-8c17-aa9bfb9106f8.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916844291,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00002-3d329e58-1eb2-465a-ba51-702884163d6a.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916858288,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00003-c2da6c8f-addf-4b50-8e24-f195803dc7b7.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916969175,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00004-9d984fc8-1528-4942-b93f-14146ba7007a.metadata.json"
+ }, {
+ "timestamp-ms" : 1660916969420,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/00005-7a1f9194-1266-4086-8113-3dc9309b7940.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/version-hint.txt
new file mode 100644
index 000000000..7f8f011eb
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files/metadata/version-hint.txt
@@ -0,0 +1 @@
+7
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155232_f36b3f3e-183a-475a-a7d1-d16898fd4524-job_16618645111570_0003-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155232_f36b3f3e-183a-475a-a7d1-d16898fd4524-job_16618645111570_0003-00001.orc
new file mode 100644
index 000000000..b12546ca3
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155232_f36b3f3e-183a-475a-a7d1-d16898fd4524-job_16618645111570_0003-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155307_610e0b78-1186-477b-a30b-9943cb4e90e9-job_16618645111570_0003-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155307_610e0b78-1186-477b-a30b-9943cb4e90e9-job_16618645111570_0003-00001.orc
new file mode 100644
index 000000000..c07c897a2
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155307_610e0b78-1186-477b-a30b-9943cb4e90e9-job_16618645111570_0003-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111570_0004-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111570_0004-00001.orc
new file mode 100644
index 000000000..fab2995a9
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111570_0004-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155411_eee97f13-df3d-408c-bcff-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155411_eee97f13-df3d-408c-bcff-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc
new file mode 100644
index 000000000..09258fa24
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-data-boroknagyz_20220830155411_eee97f13-df3d-408c-bcff-8e3b0dcf8fa7-job_16618645111570_0004-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111571_0004-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111571_0004-00001.orc
new file mode 100644
index 000000000..c57abf904
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155337_fcbda57d-f37d-48cd-afe4-28d0bad0a4c0-job_16618645111571_0004-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155430_32c848c9-37ad-4f77-a0fd-0a254d15246c-job_16618645111571_0004-00001.orc b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155430_32c848c9-37ad-4f77-a0fd-0a254d15246c-job_16618645111571_0004-00001.orc
new file mode 100644
index 000000000..a28367f6c
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/data/00000-0-delete-boroknagyz_20220830155430_32c848c9-37ad-4f77-a0fd-0a254d15246c-job_16618645111571_0004-00001.orc differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro
new file mode 100644
index 000000000..9fa676a5b
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/560427d8-9e81-4111-9b40-3aeb73af7049-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro
new file mode 100644
index 000000000..239200115
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/5f58eff5-d9a4-44eb-a208-244d0b6b927b-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro
new file mode 100644
index 000000000..7f66c8257
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/666190a5-67e0-431f-b142-7ddf9f933d4f-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/9677edab-12db-429c-aa24-5e3a112caed4-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/9677edab-12db-429c-aa24-5e3a112caed4-m0.avro
new file mode 100644
index 000000000..ff9e8fffb
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/9677edab-12db-429c-aa24-5e3a112caed4-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/a8152ff6-a39c-435a-916b-d0df02388331-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/a8152ff6-a39c-435a-916b-d0df02388331-m0.avro
new file mode 100644
index 000000000..0a2aab306
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/a8152ff6-a39c-435a-916b-d0df02388331-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro
new file mode 100644
index 000000000..4702f5d40
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/e1f383b4-0b8a-4a19-8c70-4ea23978309d-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro
new file mode 100644
index 000000000..8f8480919
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro
new file mode 100644
index 000000000..7533e12d1
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro
new file mode 100644
index 000000000..ec28d5e02
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8479298697893993480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8479298697893993480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro
new file mode 100644
index 000000000..e952c8a7b
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8479298697893993480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro
new file mode 100644
index 000000000..2c9fe8db6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro
new file mode 100644
index 000000000..8a6874081
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v1.metadata.json
new file mode 100644
index 000000000..2924ed506
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v1.metadata.json
@@ -0,0 +1,46 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1661867499625,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "write.format.default" : "orc",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v2.metadata.json
new file mode 100644
index 000000000..536de7c9b
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v2.metadata.json
@@ -0,0 +1,71 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1661867558455,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 5936629849848085185,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v3.metadata.json
new file mode 100644
index 000000000..041afe00e
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v3.metadata.json
@@ -0,0 +1,97 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1661867592515,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 8621312932233473501,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8621312932233473501,
+ "parent-snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867592515,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "339",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "679",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "snapshot-id" : 8621312932233473501
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867558455,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00001-3f96fdb7-c694-4d8e-b34c-3f682bd3a837.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v4.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v4.metadata.json
new file mode 100644
index 000000000..c60681c03
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v4.metadata.json
@@ -0,0 +1,123 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 3,
+ "last-updated-ms" : 1661867628248,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 8476486151350891395,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8621312932233473501,
+ "parent-snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867592515,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "339",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "679",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 8476486151350891395,
+ "parent-snapshot-id" : 8621312932233473501,
+ "timestamp-ms" : 1661867628248,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "336",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "1015",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "snapshot-id" : 8621312932233473501
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "snapshot-id" : 8476486151350891395
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867558455,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00001-3f96fdb7-c694-4d8e-b34c-3f682bd3a837.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00002-42184aed-2d11-4d59-a047-5ba71fd2a378.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v5.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v5.metadata.json
new file mode 100644
index 000000000..6853c5c80
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v5.metadata.json
@@ -0,0 +1,149 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 4,
+ "last-updated-ms" : 1661867628482,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 8634123667305136474,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8621312932233473501,
+ "parent-snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867592515,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "339",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "679",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 8476486151350891395,
+ "parent-snapshot-id" : 8621312932233473501,
+ "timestamp-ms" : 1661867628248,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "336",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "1015",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 8634123667305136474,
+ "parent-snapshot-id" : 8476486151350891395,
+ "timestamp-ms" : 1661867628482,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "1347",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "2362",
+ "total-data-files" : "3",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "snapshot-id" : 8621312932233473501
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "snapshot-id" : 8476486151350891395
+ }, {
+ "timestamp-ms" : 1661867628482,
+ "snapshot-id" : 8634123667305136474
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867558455,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00001-3f96fdb7-c694-4d8e-b34c-3f682bd3a837.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00002-42184aed-2d11-4d59-a047-5ba71fd2a378.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00003-6dfabfa5-497d-4d4b-badb-d6c0cfa5fd86.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v6.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v6.metadata.json
new file mode 100644
index 000000000..6e32b4ebe
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v6.metadata.json
@@ -0,0 +1,175 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 5,
+ "last-updated-ms" : 1661867657389,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 5003445199566617082,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8621312932233473501,
+ "parent-snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867592515,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "339",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "679",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 8476486151350891395,
+ "parent-snapshot-id" : 8621312932233473501,
+ "timestamp-ms" : 1661867628248,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "336",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "1015",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 8634123667305136474,
+ "parent-snapshot-id" : 8476486151350891395,
+ "timestamp-ms" : 1661867628482,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "1347",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "2362",
+ "total-data-files" : "3",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 5,
+ "snapshot-id" : 5003445199566617082,
+ "parent-snapshot-id" : 8634123667305136474,
+ "timestamp-ms" : 1661867657389,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "341",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "2703",
+ "total-data-files" : "4",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "snapshot-id" : 8621312932233473501
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "snapshot-id" : 8476486151350891395
+ }, {
+ "timestamp-ms" : 1661867628482,
+ "snapshot-id" : 8634123667305136474
+ }, {
+ "timestamp-ms" : 1661867657389,
+ "snapshot-id" : 5003445199566617082
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867558455,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00001-3f96fdb7-c694-4d8e-b34c-3f682bd3a837.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00002-42184aed-2d11-4d59-a047-5ba71fd2a378.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00003-6dfabfa5-497d-4d4b-badb-d6c0cfa5fd86.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867628482,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00004-f201703c-0958-434b-b3b9-9dbd84f151be.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v7.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v7.metadata.json
new file mode 100644
index 000000000..c231516a7
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/v7.metadata.json
@@ -0,0 +1,201 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "9e548a1f-3441-44cd-9166-5233ca6f178b",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc",
+ "last-sequence-number" : 6,
+ "last-updated-ms" : 1661867675082,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler",
+ "write.format.default" : "orc"
+ },
+ "current-snapshot-id" : 8479298697893993480,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867558455,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "340",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "340",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5936629849848085185-1-e1f383b4-0b8a-4a19-8c70-4ea23978309d.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 8621312932233473501,
+ "parent-snapshot-id" : 5936629849848085185,
+ "timestamp-ms" : 1661867592515,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "339",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "679",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8621312932233473501-1-666190a5-67e0-431f-b142-7ddf9f933d4f.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 8476486151350891395,
+ "parent-snapshot-id" : 8621312932233473501,
+ "timestamp-ms" : 1661867628248,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "1",
+ "added-files-size" : "336",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "1015",
+ "total-data-files" : "3",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8476486151350891395-1-9677edab-12db-429c-aa24-5e3a112caed4.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 4,
+ "snapshot-id" : 8634123667305136474,
+ "parent-snapshot-id" : 8476486151350891395,
+ "timestamp-ms" : 1661867628482,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "1347",
+ "added-position-deletes" : "1",
+ "changed-partition-count" : "1",
+ "total-records" : "7",
+ "total-files-size" : "2362",
+ "total-data-files" : "3",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8634123667305136474-2-a8152ff6-a39c-435a-916b-d0df02388331.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 5,
+ "snapshot-id" : 5003445199566617082,
+ "parent-snapshot-id" : 8634123667305136474,
+ "timestamp-ms" : 1661867657389,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "341",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "2703",
+ "total-data-files" : "4",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "1",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-5003445199566617082-1-5f58eff5-d9a4-44eb-a208-244d0b6b927b.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 6,
+ "snapshot-id" : 8479298697893993480,
+ "parent-snapshot-id" : 5003445199566617082,
+ "timestamp-ms" : 1661867675082,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "1367",
+ "added-position-deletes" : "3",
+ "changed-partition-count" : "1",
+ "total-records" : "10",
+ "total-files-size" : "4070",
+ "total-data-files" : "4",
+ "total-delete-files" : "2",
+ "total-position-deletes" : "4",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/snap-8479298697893993480-1-560427d8-9e81-4111-9b40-3aeb73af7049.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1661867558455,
+ "snapshot-id" : 5936629849848085185
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "snapshot-id" : 8621312932233473501
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "snapshot-id" : 8476486151350891395
+ }, {
+ "timestamp-ms" : 1661867628482,
+ "snapshot-id" : 8634123667305136474
+ }, {
+ "timestamp-ms" : 1661867657389,
+ "snapshot-id" : 5003445199566617082
+ }, {
+ "timestamp-ms" : 1661867675082,
+ "snapshot-id" : 8479298697893993480
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1661867499625,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00000-fc3a7d3b-017b-4dac-a250-e56440d4495c.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867558455,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00001-3f96fdb7-c694-4d8e-b34c-3f682bd3a837.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867592515,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00002-42184aed-2d11-4d59-a047-5ba71fd2a378.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867628248,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00003-6dfabfa5-497d-4d4b-badb-d6c0cfa5fd86.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867628482,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00004-f201703c-0958-434b-b3b9-9dbd84f151be.metadata.json"
+ }, {
+ "timestamp-ms" : 1661867657389,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/00005-03107638-513f-4b6e-b5b2-d1722f7cce0a.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/version-hint.txt
new file mode 100644
index 000000000..7f8f011eb
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc/metadata/version-hint.txt
@@ -0,0 +1 @@
+7
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155751_2ddc0706-4c90-42db-927b-d92f68b8ae1e-job_16597105613620_0030-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155751_2ddc0706-4c90-42db-927b-d92f68b8ae1e-job_16597105613620_0030-00001.parquet
new file mode 100644
index 000000000..ca6f0c4f5
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155751_2ddc0706-4c90-42db-927b-d92f68b8ae1e-job_16597105613620_0030-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613620_0030-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613620_0030-00001.parquet
new file mode 100644
index 000000000..87d3d4a22
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-data-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613620_0030-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-delete-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613621_0030-00001.parquet b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-delete-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613621_0030-00001.parquet
new file mode 100644
index 000000000..d41be4f52
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/data/00000-0-delete-boroknagyz_20220819155811_bf677db1-cb6c-451b-afc8-4013ae7eaf1f-job_16597105613621_0030-00001.parquet differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro
new file mode 100644
index 000000000..ec07b49e8
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/0f93551b-cc05-4e55-a65b-6f78e634cf4c-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro
new file mode 100644
index 000000000..7adfbd762
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/d399b245-f138-4ae2-bd1f-f82e0ee16023-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/da27d446-94df-4850-85dd-c8edcda7685e-m0.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/da27d446-94df-4850-85dd-c8edcda7685e-m0.avro
new file mode 100644
index 000000000..f998cf1c6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/da27d446-94df-4850-85dd-c8edcda7685e-m0.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-3877007445826010687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-3877007445826010687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro
new file mode 100644
index 000000000..f04ff2172
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-3877007445826010687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro
new file mode 100644
index 000000000..a42880f22
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro
new file mode 100644
index 000000000..60b675db6
Binary files /dev/null and b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro differ
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v1.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v1.metadata.json
new file mode 100644
index 000000000..a94741f2e
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v1.metadata.json
@@ -0,0 +1,45 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "b7ff3769-8a2e-440b-a018-ece76712bbc5",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows",
+ "last-sequence-number" : 0,
+ "last-updated-ms" : 1660917375159,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : -1,
+ "snapshots" : [ ],
+ "snapshot-log" : [ ],
+ "metadata-log" : [ ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v2.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v2.metadata.json
new file mode 100644
index 000000000..d3995cece
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v2.metadata.json
@@ -0,0 +1,70 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "b7ff3769-8a2e-440b-a018-ece76712bbc5",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows",
+ "last-sequence-number" : 1,
+ "last-updated-ms" : 1660917479929,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 6537343489442250314,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 6537343489442250314,
+ "timestamp-ms" : 1660917479929,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660917479929,
+ "snapshot-id" : 6537343489442250314
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660917375159,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00000-aa9b0c1a-a98f-4363-87ab-acb2be929c64.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v3.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v3.metadata.json
new file mode 100644
index 000000000..0e9189b04
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v3.metadata.json
@@ -0,0 +1,96 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "b7ff3769-8a2e-440b-a018-ece76712bbc5",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows",
+ "last-sequence-number" : 2,
+ "last-updated-ms" : 1660917497596,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 5392552459484846077,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 6537343489442250314,
+ "timestamp-ms" : 1660917479929,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5392552459484846077,
+ "parent-snapshot-id" : 6537343489442250314,
+ "timestamp-ms" : 1660917497596,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660917479929,
+ "snapshot-id" : 6537343489442250314
+ }, {
+ "timestamp-ms" : 1660917497596,
+ "snapshot-id" : 5392552459484846077
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660917375159,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00000-aa9b0c1a-a98f-4363-87ab-acb2be929c64.metadata.json"
+ }, {
+ "timestamp-ms" : 1660917479929,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00001-fd65fefa-6d6b-4c6d-90a6-fe6f8aab0053.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v4.metadata.json b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v4.metadata.json
new file mode 100644
index 000000000..1b540f5c8
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/v4.metadata.json
@@ -0,0 +1,122 @@
+{
+ "format-version" : 2,
+ "table-uuid" : "b7ff3769-8a2e-440b-a018-ece76712bbc5",
+ "location" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows",
+ "last-sequence-number" : 3,
+ "last-updated-ms" : 1660917497826,
+ "last-column-id" : 2,
+ "current-schema-id" : 0,
+ "schemas" : [ {
+ "type" : "struct",
+ "schema-id" : 0,
+ "fields" : [ {
+ "id" : 1,
+ "name" : "i",
+ "required" : false,
+ "type" : "int"
+ }, {
+ "id" : 2,
+ "name" : "s",
+ "required" : false,
+ "type" : "string"
+ } ]
+ } ],
+ "default-spec-id" : 0,
+ "partition-specs" : [ {
+ "spec-id" : 0,
+ "fields" : [ ]
+ } ],
+ "last-partition-id" : 999,
+ "default-sort-order-id" : 0,
+ "sort-orders" : [ {
+ "order-id" : 0,
+ "fields" : [ ]
+ } ],
+ "properties" : {
+ "engine.hive.enabled" : "true",
+ "bucketing_version" : "2",
+ "serialization.format" : "1",
+ "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
+ },
+ "current-snapshot-id" : 3877007445826010687,
+ "snapshots" : [ {
+ "sequence-number" : 1,
+ "snapshot-id" : 6537343489442250314,
+ "timestamp-ms" : 1660917479929,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "3",
+ "total-files-size" : "625",
+ "total-data-files" : "1",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-6537343489442250314-1-0f93551b-cc05-4e55-a65b-6f78e634cf4c.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 2,
+ "snapshot-id" : 5392552459484846077,
+ "parent-snapshot-id" : 6537343489442250314,
+ "timestamp-ms" : 1660917497596,
+ "summary" : {
+ "operation" : "append",
+ "added-data-files" : "1",
+ "added-records" : "3",
+ "added-files-size" : "625",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "1250",
+ "total-data-files" : "2",
+ "total-delete-files" : "0",
+ "total-position-deletes" : "0",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-5392552459484846077-1-d399b245-f138-4ae2-bd1f-f82e0ee16023.avro",
+ "schema-id" : 0
+ }, {
+ "sequence-number" : 3,
+ "snapshot-id" : 3877007445826010687,
+ "parent-snapshot-id" : 5392552459484846077,
+ "timestamp-ms" : 1660917497826,
+ "summary" : {
+ "operation" : "overwrite",
+ "added-delete-files" : "1",
+ "added-files-size" : "2666",
+ "added-position-deletes" : "3",
+ "changed-partition-count" : "1",
+ "total-records" : "6",
+ "total-files-size" : "3916",
+ "total-data-files" : "2",
+ "total-delete-files" : "1",
+ "total-position-deletes" : "3",
+ "total-equality-deletes" : "0"
+ },
+ "manifest-list" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/snap-3877007445826010687-2-da27d446-94df-4850-85dd-c8edcda7685e.avro",
+ "schema-id" : 0
+ } ],
+ "snapshot-log" : [ {
+ "timestamp-ms" : 1660917479929,
+ "snapshot-id" : 6537343489442250314
+ }, {
+ "timestamp-ms" : 1660917497596,
+ "snapshot-id" : 5392552459484846077
+ }, {
+ "timestamp-ms" : 1660917497826,
+ "snapshot-id" : 3877007445826010687
+ } ],
+ "metadata-log" : [ {
+ "timestamp-ms" : 1660917375159,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00000-aa9b0c1a-a98f-4363-87ab-acb2be929c64.metadata.json"
+ }, {
+ "timestamp-ms" : 1660917479929,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00001-fd65fefa-6d6b-4c6d-90a6-fe6f8aab0053.metadata.json"
+ }, {
+ "timestamp-ms" : 1660917497596,
+ "metadata-file" : "/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/00002-8c64f879-a662-4366-a812-f7703ac2b1c1.metadata.json"
+ } ]
+}
\ No newline at end of file
diff --git a/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/version-hint.txt b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/version-hint.txt
new file mode 100644
index 000000000..b8626c4cf
--- /dev/null
+++ b/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows/metadata/version-hint.txt
@@ -0,0 +1 @@
+4
diff --git a/testdata/datasets/functional/functional_schema_template.sql b/testdata/datasets/functional/functional_schema_template.sql
index d8e185d0c..c8d88f8df 100644
--- a/testdata/datasets/functional/functional_schema_template.sql
+++ b/testdata/datasets/functional/functional_schema_template.sql
@@ -3445,6 +3445,141 @@ hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/i
---- DATASET
functional
---- BASE_TABLE_NAME
+iceberg_v2_no_deletes
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_no_deletes',
+ 'format-version'='2');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_no_deletes_orc
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_no_deletes_orc',
+ 'format-version'='2', 'write.format.default'='orc');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_no_deletes_orc /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_positional_delete_all_rows
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_positional_delete_all_rows',
+ 'format-version'='2');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_positional_delete_all_rows_orc
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_positional_delete_all_rows_orc',
+ 'format-version'='2', 'write.format.default'='orc');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_delete_all_rows_orc /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_positional_not_all_data_files_have_delete_files
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_positional_not_all_data_files_have_delete_files',
+ 'format-version'='2');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_positional_not_all_data_files_have_delete_files_orc',
+ 'format-version'='2', 'write.format.default'='orc');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_not_all_data_files_have_delete_files_orc /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_positional_update_all_rows
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_positional_update_all_rows',
+ 'format-version'='2');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_positional_update_all_rows /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_partitioned_position_deletes
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_partitioned_position_deletes',
+ 'format-version'='2');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
+iceberg_v2_partitioned_position_deletes_orc
+---- CREATE
+CREATE EXTERNAL TABLE IF NOT EXISTS {db_name}{db_suffix}.{table_name}
+STORED AS ICEBERG
+TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
+ 'iceberg.catalog_location'='/test-warehouse/iceberg_test/hadoop_catalog',
+ 'iceberg.table_identifier'='ice.iceberg_v2_partitioned_position_deletes_orc',
+ 'format-version'='2', 'write.format.default'='orc');
+---- DEPENDENT_LOAD
+`hadoop fs -mkdir -p /test-warehouse/iceberg_test/hadoop_catalog/ice && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_partitioned_position_deletes_orc /test-warehouse/iceberg_test/hadoop_catalog/ice
+====
+---- DATASET
+functional
+---- BASE_TABLE_NAME
mv1_alltypes_jointbl
---- HIVE_MAJOR_VERSION
3
@@ -3566,4 +3701,4 @@ values (
map(1, cast("value1" as binary), 2, cast("value2" as binary)),
named_struct("i", 0, "b", cast("member" as binary))
)
-====
\ No newline at end of file
+====
diff --git a/testdata/datasets/functional/schema_constraints.csv b/testdata/datasets/functional/schema_constraints.csv
index 79b2c6e26..2836a6f82 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -82,6 +82,15 @@ table_name:iceberg_timestamp_part, constraint:restrict_to, table_format:parquet/
table_name:iceberg_timestamptz_part, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_uppercase_col, constraint:restrict_to, table_format:parquet/none/none
table_name:iceberg_v2_delete_positional, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_no_deletes, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_no_deletes_orc, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_positional_update_all_rows, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_positional_delete_all_rows, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_positional_delete_all_rows_orc, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_positional_not_all_data_files_have_delete_files, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_positional_not_all_data_files_have_delete_files_orc, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_partitioned_position_deletes, constraint:restrict_to, table_format:parquet/none/none
+table_name:iceberg_v2_partitioned_position_deletes_orc, constraint:restrict_to, table_format:parquet/none/none
# TODO: Support Avro. Data loading currently fails for Avro because complex types
# cannot be converted to the corresponding Avro types yet.
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test b/testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test
new file mode 100644
index 000000000..621fd7e4a
--- /dev/null
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test
@@ -0,0 +1,693 @@
+SELECT count(*) from iceberg_v2_no_deletes
+---- PLAN
+PLAN-ROOT SINK
+|
+00:UNION
+ constant-operands=1
+ row-size=8B cardinality=1
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+00:UNION
+ constant-operands=1
+ row-size=8B cardinality=1
+====
+SELECT count(*) from iceberg_v2_no_deletes where i > 1
+---- PLAN
+PLAN-ROOT SINK
+|
+01:AGGREGATE [FINALIZE]
+| output: count(*)
+| row-size=8B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_no_deletes]
+ HDFS partitions=1/1 files=1 size=625B
+ predicates: i > 1
+ row-size=4B cardinality=400
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+03:AGGREGATE [FINALIZE]
+| output: count:merge(*)
+| row-size=8B cardinality=1
+|
+02:EXCHANGE [UNPARTITIONED]
+|
+01:AGGREGATE
+| output: count(*)
+| row-size=8B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_no_deletes]
+ HDFS partitions=1/1 files=1 size=625B
+ predicates: i > 1
+ row-size=4B cardinality=400
+====
+SELECT count(*) from iceberg_v2_delete_positional;
+---- PLAN
+PLAN-ROOT SINK
+|
+03:AGGREGATE [FINALIZE]
+| output: count(*)
+| row-size=8B cardinality=1
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=20B cardinality=4.73K
+|
+|--01:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional-POSITION-DELETE-01 functional_parquet.iceberg_v2_delete_positional-position-delete]
+| HDFS partitions=1/1 files=1 size=1.54KB
+| row-size=160B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional]
+ HDFS partitions=1/1 files=1 size=662B
+ row-size=20B cardinality=4.73K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:AGGREGATE [FINALIZE]
+| output: count:merge(*)
+| row-size=8B cardinality=1
+|
+05:EXCHANGE [UNPARTITIONED]
+|
+03:AGGREGATE
+| output: count(*)
+| row-size=8B cardinality=1
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=20B cardinality=4.73K
+|
+|--04:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional-POSITION-DELETE-01 functional_parquet.iceberg_v2_delete_positional-position-delete]
+| HDFS partitions=1/1 files=1 size=1.54KB
+| row-size=160B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional]
+ HDFS partitions=1/1 files=1 size=662B
+ row-size=20B cardinality=4.73K
+====
+SELECT * from iceberg_v2_delete_positional;
+---- PLAN
+PLAN-ROOT SINK
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=40B cardinality=4.73K
+|
+|--01:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional-POSITION-DELETE-01 functional_parquet.iceberg_v2_delete_positional-position-delete]
+| HDFS partitions=1/1 files=1 size=1.54KB
+| row-size=160B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional]
+ HDFS partitions=1/1 files=1 size=662B
+ row-size=40B cardinality=4.73K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+04:EXCHANGE [UNPARTITIONED]
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=40B cardinality=4.73K
+|
+|--03:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional-POSITION-DELETE-01 functional_parquet.iceberg_v2_delete_positional-position-delete]
+| HDFS partitions=1/1 files=1 size=1.54KB
+| row-size=160B cardinality=1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_delete_positional]
+ HDFS partitions=1/1 files=1 size=662B
+ row-size=40B cardinality=4.73K
+====
+SELECT * from iceberg_v2_positional_delete_all_rows;
+---- PLAN
+PLAN-ROOT SINK
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=36B cardinality=8.93K
+|
+|--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_delete_all_rows-position-delete]
+| HDFS partitions=1/1 files=1 size=2.60KB
+| row-size=224B cardinality=3
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=8.93K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+04:EXCHANGE [UNPARTITIONED]
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=36B cardinality=8.93K
+|
+|--03:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_delete_all_rows-position-delete]
+| HDFS partitions=1/1 files=1 size=2.60KB
+| row-size=224B cardinality=3
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=8.93K
+====
+SELECT * from iceberg_v2_no_deletes limit 1
+---- PLAN
+PLAN-ROOT SINK
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_no_deletes]
+ HDFS partitions=1/1 files=1 size=625B
+ limit: 1
+ row-size=16B cardinality=1
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+01:EXCHANGE [UNPARTITIONED]
+| limit: 1
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_no_deletes]
+ HDFS partitions=1/1 files=1 size=625B
+ limit: 1
+ row-size=16B cardinality=1
+====
+SELECT * from iceberg_v2_positional_delete_all_rows limit 1
+---- PLAN
+PLAN-ROOT SINK
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| limit: 1
+| row-size=36B cardinality=1
+|
+|--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_delete_all_rows-position-delete]
+| HDFS partitions=1/1 files=1 size=2.60KB
+| row-size=224B cardinality=3
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=8.93K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+04:EXCHANGE [UNPARTITIONED]
+| limit: 1
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| limit: 1
+| row-size=36B cardinality=1
+|
+|--03:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_delete_all_rows-position-delete]
+| HDFS partitions=1/1 files=1 size=2.60KB
+| row-size=224B cardinality=3
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_positional_delete_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=8.93K
+====
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files limit 1
+---- PLAN
+PLAN-ROOT SINK
+|
+04:UNION
+| pass-through-operands: all
+| limit: 1
+| row-size=36B cardinality=1
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=34.20K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:EXCHANGE [UNPARTITIONED]
+| limit: 1
+|
+04:UNION
+| pass-through-operands: all
+| limit: 1
+| row-size=36B cardinality=1
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=34.20K
+| |
+| |--05:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+====
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files
+---- PLAN
+PLAN-ROOT SINK
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=34.20K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:EXCHANGE [UNPARTITIONED]
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=34.20K
+| |
+| |--05:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+====
+SELECT * from iceberg_v2_positional_update_all_rows
+---- PLAN
+PLAN-ROOT SINK
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=27.47K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=13.74K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_update_all_rows-position-delete]
+| | HDFS partitions=1/1 files=1 size=2.60KB
+| | row-size=224B cardinality=3
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=36B cardinality=13.74K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=13.74K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:EXCHANGE [UNPARTITIONED]
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=27.47K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=13.74K
+| |
+| |--05:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_update_all_rows-position-delete]
+| | HDFS partitions=1/1 files=1 size=2.60KB
+| | row-size=224B cardinality=3
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=36B cardinality=13.74K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+ HDFS partitions=1/1 files=1 size=625B
+ row-size=36B cardinality=13.74K
+====
+SELECT * from iceberg_v2_partitioned_position_deletes
+---- PLAN
+PLAN-ROOT SINK
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| row-size=64B cardinality=4.96K
+|
+|--01:SCAN HDFS [functional_parquet.iceberg_v2_partitioned_position_deletes-POSITION-DELETE-01 functional_parquet.iceberg_v2_partitioned_position_deletes-position-delete]
+| HDFS partitions=1/1 files=3 size=9.47KB
+| row-size=182B cardinality=10
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_partitioned_position_deletes]
+ HDFS partitions=1/1 files=3 size=3.48KB
+ row-size=64B cardinality=4.96K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+04:EXCHANGE [UNPARTITIONED]
+|
+02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| row-size=64B cardinality=4.96K
+|
+|--03:EXCHANGE [BROADCAST]
+| |
+| 01:SCAN HDFS [functional_parquet.iceberg_v2_partitioned_position_deletes-POSITION-DELETE-01 functional_parquet.iceberg_v2_partitioned_position_deletes-position-delete]
+| HDFS partitions=1/1 files=3 size=9.47KB
+| row-size=182B cardinality=10
+|
+00:SCAN HDFS [functional_parquet.iceberg_v2_partitioned_position_deletes]
+ HDFS partitions=1/1 files=3 size=3.48KB
+ row-size=64B cardinality=4.96K
+====
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files
+WHERE i > 2
+---- PLAN
+PLAN-ROOT SINK
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=6.84K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=3.42K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| predicates: i > 2
+| row-size=36B cardinality=3.42K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ predicates: i > 2
+ row-size=36B cardinality=3.42K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:EXCHANGE [UNPARTITIONED]
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=6.84K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=3.42K
+| |
+| |--05:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| predicates: i > 2
+| row-size=36B cardinality=3.42K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ predicates: i > 2
+ row-size=36B cardinality=3.42K
+====
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files for system_version as of 1497619269847778439
+minus
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files for system_version as of 4363979609026842966;
+---- PLAN
+PLAN-ROOT SINK
+|
+07:HASH JOIN [LEFT ANTI JOIN]
+| hash predicates: functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i IS NOT DISTINCT FROM functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i, functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s IS NOT DISTINCT FROM functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s
+| row-size=16B cardinality=68.39K
+|
+|--06:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=16B cardinality=34.20K
+|
+05:AGGREGATE [FINALIZE]
+| group by: functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i, functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s
+| row-size=16B cardinality=68.39K
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=34.20K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+12:EXCHANGE [UNPARTITIONED]
+|
+07:HASH JOIN [LEFT ANTI JOIN, PARTITIONED]
+| hash predicates: functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i IS NOT DISTINCT FROM functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i, functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s IS NOT DISTINCT FROM functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s
+| row-size=16B cardinality=68.39K
+|
+|--11:EXCHANGE [HASH(functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i,functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s)]
+| |
+| 06:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=16B cardinality=34.20K
+|
+10:AGGREGATE [FINALIZE]
+| group by: functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i, functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s
+| row-size=16B cardinality=68.39K
+|
+09:EXCHANGE [HASH(functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i,functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s)]
+|
+05:AGGREGATE [STREAMING]
+| group by: functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.i, functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files.s
+| row-size=16B cardinality=68.39K
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=34.20K
+| |
+| |--08:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ row-size=36B cardinality=34.20K
+====
+with v as (select i + 1000 as ii, upper(s) as ss from iceberg_v2_positional_not_all_data_files_have_delete_files)
+select * from v where ii > 1003;
+---- PLAN
+PLAN-ROOT SINK
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=6.84K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=3.42K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| predicates: i + 1000 > 1003
+| row-size=36B cardinality=3.42K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ predicates: i + 1000 > 1003
+ row-size=36B cardinality=3.42K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+06:EXCHANGE [UNPARTITIONED]
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=6.84K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=3.42K
+| |
+| |--05:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| predicates: i + 1000 > 1003
+| row-size=36B cardinality=3.42K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ predicates: i + 1000 > 1003
+ row-size=36B cardinality=3.42K
+====
+select *
+from iceberg_v2_positional_not_all_data_files_have_delete_files
+where i = (select max(i) from iceberg_v2_positional_update_all_rows);
+---- PLAN
+PLAN-ROOT SINK
+|
+11:HASH JOIN [LEFT SEMI JOIN]
+| hash predicates: i = max(i)
+| runtime filters: RF000 <- max(i)
+| row-size=36B cardinality=68.39K
+|
+|--10:AGGREGATE [FINALIZE]
+| | output: max(i)
+| | row-size=4B cardinality=1
+| |
+| 09:UNION
+| | pass-through-operands: all
+| | row-size=24B cardinality=27.47K
+| |
+| |--07:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | | row-size=24B cardinality=13.74K
+| | |
+| | |--06:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows-POSITION-DELETE-06 functional_parquet.iceberg_v2_positional_update_all_rows-position-delete]
+| | | HDFS partitions=1/1 files=1 size=2.60KB
+| | | row-size=224B cardinality=3
+| | |
+| | 05:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| | HDFS partitions=1/1 files=1 size=625B
+| | row-size=24B cardinality=13.74K
+| |
+| 08:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=24B cardinality=13.74K
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN]
+| | row-size=36B cardinality=34.20K
+| |
+| |--01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| runtime filters: RF000 -> i
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ runtime filters: RF000 -> i
+ row-size=36B cardinality=34.20K
+---- DISTRIBUTEDPLAN
+PLAN-ROOT SINK
+|
+17:EXCHANGE [UNPARTITIONED]
+|
+11:HASH JOIN [LEFT SEMI JOIN, BROADCAST]
+| hash predicates: i = max(i)
+| runtime filters: RF000 <- max(i)
+| row-size=36B cardinality=68.39K
+|
+|--16:EXCHANGE [BROADCAST]
+| |
+| 15:AGGREGATE [FINALIZE]
+| | output: max:merge(i)
+| | row-size=4B cardinality=1
+| |
+| 14:EXCHANGE [UNPARTITIONED]
+| |
+| 10:AGGREGATE
+| | output: max(i)
+| | row-size=4B cardinality=1
+| |
+| 09:UNION
+| | pass-through-operands: all
+| | row-size=24B cardinality=27.47K
+| |
+| |--07:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | | row-size=24B cardinality=13.74K
+| | |
+| | |--13:EXCHANGE [BROADCAST]
+| | | |
+| | | 06:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows-POSITION-DELETE-06 functional_parquet.iceberg_v2_positional_update_all_rows-position-delete]
+| | | HDFS partitions=1/1 files=1 size=2.60KB
+| | | row-size=224B cardinality=3
+| | |
+| | 05:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| | HDFS partitions=1/1 files=1 size=625B
+| | row-size=24B cardinality=13.74K
+| |
+| 08:SCAN HDFS [functional_parquet.iceberg_v2_positional_update_all_rows]
+| HDFS partitions=1/1 files=1 size=625B
+| row-size=24B cardinality=13.74K
+|
+04:UNION
+| pass-through-operands: all
+| row-size=36B cardinality=68.39K
+|
+|--02:DELETE EVENTS HASH JOIN [LEFT ANTI JOIN, BROADCAST]
+| | row-size=36B cardinality=34.20K
+| |
+| |--12:EXCHANGE [BROADCAST]
+| | |
+| | 01:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-POSITION-DELETE-01 functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files-position-delete]
+| | HDFS partitions=1/1 files=2 size=5.33KB
+| | row-size=245B cardinality=4
+| |
+| 00:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+| HDFS partitions=1/1 files=2 size=1.22KB
+| runtime filters: RF000 -> i
+| row-size=36B cardinality=34.20K
+|
+03:SCAN HDFS [functional_parquet.iceberg_v2_positional_not_all_data_files_have_delete_files]
+ HDFS partitions=1/1 files=2 size=1.22KB
+ runtime filters: RF000 -> i
+ row-size=36B cardinality=34.20K
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
index bbf62be8e..db8f0c57b 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
@@ -641,11 +641,6 @@ ALTER TABLE iceberg_alter_part SET PARTITION SPEC (HOUR(d));
ImpalaRuntimeException: Failed to ALTER table 'iceberg_alter_part': Cannot partition type date by hour
====
---- QUERY
-select * from functional_parquet.iceberg_v2_delete_positional;
----- CATCH
-row_regex:.*CAUSED BY: TableLoadingException: Unsupported Iceberg V2 feature, table .* contains delete files..*
-====
----- QUERY
CREATE TABLE clone_ice LIKE functional_parquet.alltypestiny STORED AS ICEBERG;
---- CATCH
functional_parquet.alltypestiny cannot be cloned into an Iceberg table because it is not an Iceberg table.
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
new file mode 100644
index 000000000..3fb363311
--- /dev/null
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
@@ -0,0 +1,201 @@
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_no_deletes_orc
+---- RESULTS
+3
+---- TYPES
+bigint
+---- RUNTIME_PROFILE
+aggregation(SUM, NumRowGroups): 0
+aggregation(SUM, NumFileMetadataRead): 0
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+---- RESULTS
+6
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_partitioned_position_deletes_orc
+---- RESULTS
+10
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_no_deletes_orc where i = 2;
+---- RESULTS
+1
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_delete_all_rows where i > 2;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+where i < 4;
+---- RESULTS
+3
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_partitioned_position_deletes_orc
+where action = 'click';
+---- RESULTS
+3
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT * from iceberg_v2_no_deletes_orc
+---- RESULTS
+1,'x'
+2,'y'
+3,'z'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+---- RESULTS
+1,'a'
+2,'b'
+3,'c'
+5,'X'
+4,'d'
+6,'f'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_partitioned_position_deletes_orc
+---- RESULTS
+6,'Alex','view',2020-01-01 09:00:00
+20,'Alex','view',2020-01-01 09:00:00
+4,'Alex','view',2020-01-01 09:00:00
+18,'Alan','click',2020-01-01 10:00:00
+12,'Alan','click',2020-01-01 10:00:00
+10,'Alan','click',2020-01-01 10:00:00
+2,'Lisa','download',2020-01-01 11:00:00
+14,'Lisa','download',2020-01-01 11:00:00
+8,'Lisa','download',2020-01-01 11:00:00
+16,'Lisa','download',2020-01-01 11:00:00
+---- TYPES
+INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_no_deletes_orc
+---- RESULTS
+'AGYZ_20220830153046_7C9AAA51-D601-4C5B-A51C-67C1E9E1FE29-JOB_16618645111570_0001-00001.ORC',1000,1,'x'
+'AGYZ_20220830153046_7C9AAA51-D601-4C5B-A51C-67C1E9E1FE29-JOB_16618645111570_0001-00001.ORC',1001,2,'y'
+'AGYZ_20220830153046_7C9AAA51-D601-4C5B-A51C-67C1E9E1FE29-JOB_16618645111570_0001-00001.ORC',1002,3,'z'
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+---- RESULTS
+'AGYZ_20220830155337_FCBDA57D-F37D-48CD-AFE4-28D0BAD0A4C0-JOB_16618645111570_0004-00001.ORC',1000,5,'X'
+'AGYZ_20220830155232_F36B3F3E-183A-475A-A7D1-D16898FD4524-JOB_16618645111570_0003-00001.ORC',1000,1,'a'
+'AGYZ_20220830155232_F36B3F3E-183A-475A-A7D1-D16898FD4524-JOB_16618645111570_0003-00001.ORC',1001,2,'b'
+'AGYZ_20220830155232_F36B3F3E-183A-475A-A7D1-D16898FD4524-JOB_16618645111570_0003-00001.ORC',1002,3,'c'
+'AGYZ_20220830155307_610E0B78-1186-477B-A30B-9943CB4E90E9-JOB_16618645111570_0003-00001.ORC',1000,4,'d'
+'AGYZ_20220830155307_610E0B78-1186-477B-A30B-9943CB4E90E9-JOB_16618645111570_0003-00001.ORC',1002,6,'f'
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_partitioned_position_deletes_orc
+---- RESULTS
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00001.ORC',1001,12,'Alan','click',2020-01-01 10:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00001.ORC',1003,18,'Alan','click',2020-01-01 10:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00001.ORC',1005,10,'Alan','click',2020-01-01 10:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00002.ORC',1000,16,'Lisa','download',2020-01-01 11:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00002.ORC',1002,2,'Lisa','download',2020-01-01 11:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00002.ORC',1004,8,'Lisa','download',2020-01-01 11:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00002.ORC',1005,14,'Lisa','download',2020-01-01 11:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00003.ORC',1002,20,'Alex','view',2020-01-01 09:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00003.ORC',1004,6,'Alex','view',2020-01-01 09:00:00
+'AGYZ_20220830173852_560D6F0E-8508-42D6-95A7-67BFA860A28C-JOB_16618645111571_0006-00003.ORC',1007,4,'Alex','view',2020-01-01 09:00:00
+---- TYPES
+STRING, BIGINT, INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+SELECT * from iceberg_v2_no_deletes_orc where i = 2;
+---- RESULTS
+2,'y'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_delete_all_rows where i > 2;
+---- RESULTS
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files_orc
+where i < 4;
+---- RESULTS
+1,'a'
+2,'b'
+3,'c'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_partitioned_position_deletes_orc
+where action = 'click';
+---- RESULTS
+18,'Alan','click',2020-01-01 10:00:00
+12,'Alan','click',2020-01-01 10:00:00
+10,'Alan','click',2020-01-01 10:00:00
+---- TYPES
+INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+SELECT action, count(*) from iceberg_v2_partitioned_position_deletes_orc
+group by action;
+---- RESULTS
+'click',3
+'download',4
+'view',3
+---- TYPES
+STRING, BIGINT
+====
+---- QUERY
+with v as (select i + 1000 as ii, upper(s) as ss from iceberg_v2_positional_not_all_data_files_have_delete_files_orc)
+select * from v where ii > 1003;
+---- RESULTS
+1005,'X'
+1004,'D'
+1006,'F'
+---- TYPES
+BIGINT, STRING
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test b/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
new file mode 100644
index 000000000..a8d2fc63c
--- /dev/null
+++ b/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
@@ -0,0 +1,340 @@
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_no_deletes
+---- RESULTS
+3
+---- TYPES
+bigint
+---- RUNTIME_PROFILE
+aggregation(SUM, NumRowGroups): 0
+aggregation(SUM, NumFileMetadataRead): 0
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_delete_positional;
+---- RESULTS
+2
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_not_all_data_files_have_delete_files
+---- RESULTS
+6
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_update_all_rows
+---- RESULTS
+3
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_partitioned_position_deletes
+---- RESULTS
+10
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_no_deletes where i = 2;
+---- RESULTS
+1
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_delete_positional where id = 2;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_delete_all_rows where i > 2;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_not_all_data_files_have_delete_files
+where i < 4;
+---- RESULTS
+3
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_positional_update_all_rows
+where i = 3;
+---- RESULTS
+1
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT count(*) from iceberg_v2_partitioned_position_deletes
+where action = 'click';
+---- RESULTS
+3
+---- TYPES
+bigint
+====
+---- QUERY
+SELECT * from iceberg_v2_no_deletes
+---- RESULTS
+1,'x'
+2,'y'
+3,'z'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_delete_positional;
+---- RESULTS
+1,'a'
+3,'c'
+---- TYPES
+BIGINT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files
+---- RESULTS
+1,'a'
+2,'b'
+3,'c'
+5,'X'
+4,'d'
+6,'f'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_update_all_rows
+---- RESULTS
+1,'A'
+2,'B'
+3,'C'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_partitioned_position_deletes
+---- RESULTS
+6,'Alex','view',2020-01-01 09:00:00
+20,'Alex','view',2020-01-01 09:00:00
+4,'Alex','view',2020-01-01 09:00:00
+18,'Alan','click',2020-01-01 10:00:00
+12,'Alan','click',2020-01-01 10:00:00
+10,'Alan','click',2020-01-01 10:00:00
+2,'Lisa','download',2020-01-01 11:00:00
+14,'Lisa','download',2020-01-01 11:00:00
+8,'Lisa','download',2020-01-01 11:00:00
+16,'Lisa','download',2020-01-01 11:00:00
+---- TYPES
+INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_no_deletes
+---- RESULTS
+'_20220819180420_A7E5A731-8762-4B59-B3F2-FE6F065CF59B-JOB_16597105613620_0031-00001.PARQUET',1000,1,'x'
+'_20220819180420_A7E5A731-8762-4B59-B3F2-FE6F065CF59B-JOB_16597105613620_0031-00001.PARQUET',1001,2,'y'
+'_20220819180420_A7E5A731-8762-4B59-B3F2-FE6F065CF59B-JOB_16597105613620_0031-00001.PARQUET',1002,3,'z'
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_delete_positional;
+---- RESULTS
+'EBERG_V2_DELETE_POSITIONAL/DATA/00000-0-FB178C51-B12A-4C5F-A66E-A8E9375DAEBA-00001.PARQUET',1000,1,'a'
+'EBERG_V2_DELETE_POSITIONAL/DATA/00000-0-FB178C51-B12A-4C5F-A66E-A8E9375DAEBA-00001.PARQUET',1002,3,'c'
+---- TYPES
+STRING, BIGINT, BIGINT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_positional_delete_all_rows;
+---- RESULTS
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_positional_not_all_data_files_have_delete_files
+---- RESULTS
+'_20220819154646_1CAD8C38-C65E-4C7C-B516-C4D9FAF82448-JOB_16597105613620_0026-00001.PARQUET',1000,1,'a'
+'_20220819154646_1CAD8C38-C65E-4C7C-B516-C4D9FAF82448-JOB_16597105613620_0026-00001.PARQUET',1001,2,'b'
+'_20220819154646_1CAD8C38-C65E-4C7C-B516-C4D9FAF82448-JOB_16597105613620_0026-00001.PARQUET',1002,3,'c'
+'_20220819154922_62429D29-6C44-4707-B348-AC189B8D79D3-JOB_16597105613620_0026-00001.PARQUET',1000,5,'X'
+'_20220819154718_DB95AEAE-D530-4FBA-8336-E47FA712B987-JOB_16597105613620_0026-00001.PARQUET',1000,4,'d'
+'_20220819154718_DB95AEAE-D530-4FBA-8336-E47FA712B987-JOB_16597105613620_0026-00001.PARQUET',1002,6,'f'
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_positional_update_all_rows
+---- RESULTS
+'_20220819155811_BF677DB1-CB6C-451B-AFC8-4013AE7EAF1F-JOB_16597105613620_0030-00001.PARQUET',1000,1,'A'
+'_20220819155811_BF677DB1-CB6C-451B-AFC8-4013AE7EAF1F-JOB_16597105613620_0030-00001.PARQUET',1001,2,'B'
+'_20220819155811_BF677DB1-CB6C-451B-AFC8-4013AE7EAF1F-JOB_16597105613620_0030-00001.PARQUET',1002,3,'C'
+---- TYPES
+STRING, BIGINT, INT, STRING
+====
+---- QUERY
+SELECT strright(upper(input__file__name),90), file__position + 1000, * from iceberg_v2_partitioned_position_deletes
+---- RESULTS
+'POSITION_DELETES/DATA/ACTION=VIEW/874B32D9A15DA206-F60E01CB00000004_1711435901_DATA.0.PARQ',1000,6,'Alex','view',2020-01-01 09:00:00
+'POSITION_DELETES/DATA/ACTION=VIEW/874B32D9A15DA206-F60E01CB00000004_1711435901_DATA.0.PARQ',1003,20,'Alex','view',2020-01-01 09:00:00
+'POSITION_DELETES/DATA/ACTION=VIEW/874B32D9A15DA206-F60E01CB00000004_1711435901_DATA.0.PARQ',1007,4,'Alex','view',2020-01-01 09:00:00
+'TION_DELETES/DATA/ACTION=DOWNLOAD/874B32D9A15DA206-F60E01CB00000003_1489587766_DATA.0.PARQ',1000,2,'Lisa','download',2020-01-01 11:00:00
+'TION_DELETES/DATA/ACTION=DOWNLOAD/874B32D9A15DA206-F60E01CB00000003_1489587766_DATA.0.PARQ',1002,14,'Lisa','download',2020-01-01 11:00:00
+'TION_DELETES/DATA/ACTION=DOWNLOAD/874B32D9A15DA206-F60E01CB00000003_1489587766_DATA.0.PARQ',1003,8,'Lisa','download',2020-01-01 11:00:00
+'TION_DELETES/DATA/ACTION=DOWNLOAD/874B32D9A15DA206-F60E01CB00000003_1489587766_DATA.0.PARQ',1005,16,'Lisa','download',2020-01-01 11:00:00
+'OSITION_DELETES/DATA/ACTION=CLICK/874B32D9A15DA206-F60E01CB00000003_1034098606_DATA.0.PARQ',1001,18,'Alan','click',2020-01-01 10:00:00
+'OSITION_DELETES/DATA/ACTION=CLICK/874B32D9A15DA206-F60E01CB00000003_1034098606_DATA.0.PARQ',1003,12,'Alan','click',2020-01-01 10:00:00
+'OSITION_DELETES/DATA/ACTION=CLICK/874B32D9A15DA206-F60E01CB00000003_1034098606_DATA.0.PARQ',1005,10,'Alan','click',2020-01-01 10:00:00
+---- TYPES
+STRING, BIGINT, INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+SELECT * from iceberg_v2_no_deletes where i = 2;
+---- RESULTS
+2,'y'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_delete_positional where id = 2;
+---- RESULTS
+---- TYPES
+BIGINT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_delete_all_rows where i > 2;
+---- RESULTS
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_not_all_data_files_have_delete_files
+where i < 4;
+---- RESULTS
+1,'a'
+2,'b'
+3,'c'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_positional_update_all_rows
+where i = 3;
+---- RESULTS
+3,'C'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT * from iceberg_v2_partitioned_position_deletes
+where action = 'click';
+---- RESULTS
+18,'Alan','click',2020-01-01 10:00:00
+12,'Alan','click',2020-01-01 10:00:00
+10,'Alan','click',2020-01-01 10:00:00
+---- TYPES
+INT, STRING, STRING, TIMESTAMP
+====
+---- QUERY
+select count(*)
+from iceberg_v2_positional_not_all_data_files_have_delete_files a,
+ iceberg_v2_positional_update_all_rows b
+where a.i = b.i;
+---- RESULTS
+3
+---- TYPES
+BIGINT
+====
+---- QUERY
+select *
+from iceberg_v2_positional_not_all_data_files_have_delete_files a,
+ iceberg_v2_positional_update_all_rows b
+where a.i = b.i;
+---- RESULTS
+1,'a',1,'A'
+2,'b',2,'B'
+3,'c',3,'C'
+---- TYPES
+INT, STRING, INT, STRING
+====
+---- QUERY
+select *
+from iceberg_v2_positional_not_all_data_files_have_delete_files
+where i = (select max(i) from iceberg_v2_positional_update_all_rows);
+---- RESULTS
+3,'c'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+SELECT action, count(*) from iceberg_v2_partitioned_position_deletes
+group by action;
+---- RESULTS
+'click',3
+'download',4
+'view',3
+---- TYPES
+STRING, BIGINT
+====
+---- QUERY
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files
+union all
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files for system_version as of 5762682948883272650
+---- RESULTS
+4,'d'
+5,'e'
+6,'f'
+1,'a'
+2,'b'
+3,'c'
+5,'X'
+4,'d'
+6,'f'
+1,'a'
+2,'b'
+3,'c'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files for system_version as of 1497619269847778439
+minus
+select * from iceberg_v2_positional_not_all_data_files_have_delete_files for system_version as of 4363979609026842966;
+---- RESULTS
+5,'X'
+4,'d'
+6,'f'
+---- TYPES
+INT, STRING
+====
+---- QUERY
+with v as (select i + 1000 as ii, upper(s) as ss from iceberg_v2_positional_not_all_data_files_have_delete_files)
+select * from v where ii > 1003;
+---- RESULTS
+1005,'X'
+1004,'D'
+1006,'F'
+---- TYPES
+BIGINT, STRING
+====
diff --git a/tests/query_test/test_iceberg.py b/tests/query_test/test_iceberg.py
index d1995f9a7..1cc9214b6 100644
--- a/tests/query_test/test_iceberg.py
+++ b/tests/query_test/test_iceberg.py
@@ -31,7 +31,7 @@ import json
from tests.beeswax.impala_beeswax import ImpalaBeeswaxException
from tests.common.iceberg_test_suite import IcebergTestSuite
-from tests.common.skip import SkipIf
+from tests.common.skip import SkipIf, SkipIfDockerizedCluster
from tests.common.file_utils import create_iceberg_table_from_directory
from tests.shell.util import run_impala_shell_cmd
from tests.util.filesystem_utils import get_fs_path, IS_HDFS
@@ -779,3 +779,30 @@ class TestIcebergTable(IcebergTestSuite):
args = ['-q', 'DROP TABLE {0}.{1}'.format(db_name, tbl_name)]
results = run_impala_shell_cmd(vector, args)
+
+
+class TestIcebergV2Table(IcebergTestSuite):
+ """Tests related to Iceberg V2 tables."""
+
+ @classmethod
+ def get_workload(cls):
+ return 'functional-query'
+
+ @classmethod
+ def add_test_dimensions(cls):
+ super(TestIcebergV2Table, cls).add_test_dimensions()
+ cls.ImpalaTestMatrix.add_constraint(
+ lambda v: v.get_value('table_format').file_format == 'parquet')
+
+ # The test uses pre-written Iceberg tables where the position delete files refer to
+ # the data files via full URI, i.e. they start with 'hdfs://localhost:2050/...'. In the
+ # dockerised environment the namenode is accessible on a different hostname/port.
+ @SkipIfDockerizedCluster.internal_hostname
+ @SkipIf.not_hdfs
+ def test_read_position_deletes(self, vector):
+ self.run_test_case('QueryTest/iceberg-v2-read-position-deletes', vector)
+
+ @SkipIfDockerizedCluster.internal_hostname
+ @SkipIf.not_hdfs
+ def test_read_position_deletes_orc(self, vector):
+ self.run_test_case('QueryTest/iceberg-v2-read-position-deletes-orc', vector)
\ No newline at end of file
[impala] 05/05: IMPALA-11525: Rename Exec libraries to avoid conflicts with external libraries
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit d74d6994cc25ed2090886d6b406cf477a6ccf6b4
Author: Peter Rozsa <pr...@cloudera.com>
AuthorDate: Thu Aug 25 18:51:32 2022 +0200
IMPALA-11525: Rename Exec libraries to avoid conflicts with external libraries
This patch renames the Exec libraries to distinguish them from
external libraries like avro, kudu_client and orc
Due to IMPALA-11257, the cmake_minimum_required calls are
also removed from the new libraries
Change-Id: I042478e371049a7aed74f43c2bbfcb7900a53dd0
Reviewed-on: http://gerrit.cloudera.org:8080/18910
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/CMakeLists.txt | 36 ++++++++++++++++++------------------
be/src/codegen/CMakeLists.txt | 4 ++--
be/src/exec/avro/CMakeLists.txt | 10 ++++------
be/src/exec/hbase/CMakeLists.txt | 6 ++----
be/src/exec/kudu/CMakeLists.txt | 6 ++----
be/src/exec/orc/CMakeLists.txt | 6 ++----
be/src/exec/parquet/CMakeLists.txt | 8 ++++----
be/src/exec/rcfile/CMakeLists.txt | 6 ++----
be/src/exec/sequence/CMakeLists.txt | 6 ++----
be/src/exec/text/CMakeLists.txt | 6 ++----
10 files changed, 40 insertions(+), 54 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 4ed427037..fa8341feb 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -438,15 +438,15 @@ set (IMPALA_LIBS
Catalog
CodeGen
Common
- Avro
- HBase
- Kudu
- Orc
- Parquet
- Rcfile
- Sequence
- Text
Exec
+ ExecAvro
+ ExecHBase
+ ExecKudu
+ ExecOrc
+ ExecParquet
+ ExecRcfile
+ ExecSequence
+ ExecText
Exprs
GlobalFlags
histogram_proto
@@ -499,11 +499,11 @@ set (UNIFIED_TEST_LIBS
CodeGenTests
CommonTests
ExecTests
- AvroTests
+ ExecAvroTests
+ ExecParquetTests
ExprsTests
GUtilTests
IoTests
- ParquetTests
RpcTests
RuntimeTests
SchedulingTests
@@ -528,15 +528,15 @@ if (BUILD_SHARED_LIBS)
BufferPool
Io
Runtime
- Avro
- HBase
- Kudu
- Orc
- Parquet
- Rcfile
- Sequence
- Text
Exec
+ ExecAvro
+ ExecHBase
+ ExecKudu
+ ExecOrc
+ ExecParquet
+ ExecRcfile
+ ExecSequence
+ ExecText
CodeGen
Exprs
Rpc
diff --git a/be/src/codegen/CMakeLists.txt b/be/src/codegen/CMakeLists.txt
index 700b4eaa9..92999b91a 100644
--- a/be/src/codegen/CMakeLists.txt
+++ b/be/src/codegen/CMakeLists.txt
@@ -85,7 +85,7 @@ add_custom_command(
${CLANG_INCLUDE_FLAGS} ${IR_INPUT_FILES} -o ${IR_TMP_OUTPUT_FILE}
COMMAND ${LLVM_OPT_EXECUTABLE} ${LLVM_OPT_IR_FLAGS} < ${IR_TMP_OUTPUT_FILE} > ${IR_OUTPUT_FILE}
COMMAND rm ${IR_TMP_OUTPUT_FILE}
- DEPENDS Exec Exprs Runtime Udf Util ${IR_INPUT_FILES}
+ DEPENDS Exec ExecAvro ExecKudu Exprs Runtime Udf Util ${IR_INPUT_FILES}
)
add_custom_command(
@@ -95,7 +95,7 @@ add_custom_command(
COMMAND ${LLVM_OPT_EXECUTABLE} ${LLVM_OPT_IR_FLAGS} < ${LEGACY_AVX_IR_TMP_OUTPUT_FILE}
> ${LEGACY_AVX_IR_OUTPUT_FILE}
COMMAND rm ${LEGACY_AVX_IR_TMP_OUTPUT_FILE}
- DEPENDS Exec Exprs Runtime Udf Util ${IR_INPUT_FILES}
+ DEPENDS Exec ExecAvro ExecKudu Exprs Runtime Udf Util ${IR_INPUT_FILES}
)
add_custom_target(compile_to_ir DEPENDS ${IR_OUTPUT_FILE})
diff --git a/be/src/exec/avro/CMakeLists.txt b/be/src/exec/avro/CMakeLists.txt
index 02e253a24..1f4bfd233 100644
--- a/be/src/exec/avro/CMakeLists.txt
+++ b/be/src/exec/avro/CMakeLists.txt
@@ -15,24 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/avro")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/avro")
-add_library(Avro
+add_library(ExecAvro
hdfs-avro-scanner.cc
hdfs-avro-scanner-ir.cc
)
-add_dependencies(Avro gen-deps)
+add_dependencies(ExecAvro gen-deps)
-add_library(AvroTests STATIC
+add_library(ExecAvroTests STATIC
hdfs-avro-scanner-test.cc
)
-add_dependencies(AvroTests gen-deps)
+add_dependencies(ExecAvroTests gen-deps)
ADD_UNIFIED_BE_LSAN_TEST(hdfs-avro-scanner-test HdfsAvroScannerTest.*)
\ No newline at end of file
diff --git a/be/src/exec/hbase/CMakeLists.txt b/be/src/exec/hbase/CMakeLists.txt
index c2e8dc939..2dc73d3df 100644
--- a/be/src/exec/hbase/CMakeLists.txt
+++ b/be/src/exec/hbase/CMakeLists.txt
@@ -15,19 +15,17 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/hbase")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/hbase")
-add_library(HBase
+add_library(ExecHBase
hbase-scan-node.cc
hbase-table-scanner.cc
hbase-table-sink.cc
hbase-table-writer.cc
)
-add_dependencies(HBase gen-deps)
\ No newline at end of file
+add_dependencies(ExecHBase gen-deps)
\ No newline at end of file
diff --git a/be/src/exec/kudu/CMakeLists.txt b/be/src/exec/kudu/CMakeLists.txt
index 728922c71..befc7a738 100644
--- a/be/src/exec/kudu/CMakeLists.txt
+++ b/be/src/exec/kudu/CMakeLists.txt
@@ -15,15 +15,13 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/kudu")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/kudu")
-add_library(Kudu
+add_library(ExecKudu
kudu-scanner.cc
kudu-scan-node.cc
kudu-scan-node-base.cc
@@ -33,4 +31,4 @@ add_library(Kudu
kudu-util-ir.cc
)
-add_dependencies(Kudu gen-deps)
\ No newline at end of file
+add_dependencies(ExecKudu gen-deps)
\ No newline at end of file
diff --git a/be/src/exec/orc/CMakeLists.txt b/be/src/exec/orc/CMakeLists.txt
index 5a3a4580d..4d06f0609 100644
--- a/be/src/exec/orc/CMakeLists.txt
+++ b/be/src/exec/orc/CMakeLists.txt
@@ -15,18 +15,16 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/orc")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/orc")
-add_library(Orc
+add_library(ExecOrc
orc-column-readers.cc
orc-metadata-utils.cc
hdfs-orc-scanner.cc
)
-add_dependencies(Orc gen-deps)
\ No newline at end of file
+add_dependencies(ExecOrc gen-deps)
\ No newline at end of file
diff --git a/be/src/exec/parquet/CMakeLists.txt b/be/src/exec/parquet/CMakeLists.txt
index f2c5b55f0..ba85b405a 100644
--- a/be/src/exec/parquet/CMakeLists.txt
+++ b/be/src/exec/parquet/CMakeLists.txt
@@ -21,7 +21,7 @@ set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/parquet")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/parquet")
-add_library(Parquet
+add_library(ExecParquet
hdfs-parquet-scanner.cc
hdfs-parquet-table-writer.cc
parquet-bloom-filter-util.cc
@@ -39,9 +39,9 @@ add_library(Parquet
parquet-struct-column-reader.cc
)
-add_dependencies(Parquet gen-deps)
+add_dependencies(ExecParquet gen-deps)
-add_library(ParquetTests STATIC
+add_library(ExecParquetTests STATIC
hdfs-parquet-scanner-test.cc
parquet-bool-decoder-test.cc
parquet-common-test.cc
@@ -50,7 +50,7 @@ add_library(ParquetTests STATIC
parquet-version-test.cc
serialize-single-value-test.cc
)
-add_dependencies(ParquetTests gen-deps)
+add_dependencies(ExecParquetTests gen-deps)
ADD_UNIFIED_BE_LSAN_TEST(parquet-bool-decoder-test ParquetBoolDecoder.*)
ADD_UNIFIED_BE_LSAN_TEST(parquet-common-test ParquetCommon.*)
diff --git a/be/src/exec/rcfile/CMakeLists.txt b/be/src/exec/rcfile/CMakeLists.txt
index b64c5c39a..3d11bf25e 100644
--- a/be/src/exec/rcfile/CMakeLists.txt
+++ b/be/src/exec/rcfile/CMakeLists.txt
@@ -15,16 +15,14 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/rcfile")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/rcfile")
-add_library(Rcfile
+add_library(ExecRcfile
hdfs-rcfile-scanner.cc
)
-add_dependencies(Rcfile gen-deps)
\ No newline at end of file
+add_dependencies(ExecRcfile gen-deps)
\ No newline at end of file
diff --git a/be/src/exec/sequence/CMakeLists.txt b/be/src/exec/sequence/CMakeLists.txt
index f6f528ab2..e6f0b73a3 100644
--- a/be/src/exec/sequence/CMakeLists.txt
+++ b/be/src/exec/sequence/CMakeLists.txt
@@ -15,16 +15,14 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/sequence")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/sequence")
-add_library(Sequence
+add_library(ExecSequence
hdfs-sequence-scanner.cc
)
-add_dependencies(Sequence gen-deps)
\ No newline at end of file
+add_dependencies(ExecSequence gen-deps)
\ No newline at end of file
diff --git a/be/src/exec/text/CMakeLists.txt b/be/src/exec/text/CMakeLists.txt
index 76ed4fd74..61e7ceaaf 100644
--- a/be/src/exec/text/CMakeLists.txt
+++ b/be/src/exec/text/CMakeLists.txt
@@ -15,17 +15,15 @@
# specific language governing permissions and limitations
# under the License.
-cmake_minimum_required(VERSION 2.6)
-
# where to put generated libraries
set(LIBRARY_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/text")
# where to put generated binaries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}/exec/text")
-add_library(Text
+add_library(ExecText
hdfs-text-scanner.cc
hdfs-plugin-text-scanner.cc
)
-add_dependencies(Text gen-deps)
\ No newline at end of file
+add_dependencies(ExecText gen-deps)
\ No newline at end of file
[impala] 01/05: IMPALA-11467: Force Boost to use /dev/random for UUID generation
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 85c6366d998a23fd940a72218eb39db57bc01002
Author: zhangyifan27 <ch...@163.com>
AuthorDate: Tue Aug 2 11:31:04 2022 +0800
IMPALA-11467: Force Boost to use /dev/random for UUID generation
To avoid crash issue at OS with kernel lower than Linux 3.17, this patch
pick kudu's solution 35b5664f908cd1250c9f01e5dff77b653cfd12b7 into impala.
Testing:
- run pre-built impalad at CentOS 7 kernel 3.10 with no crash.
Change-Id: Ic48bd59b0a846bcb91a6faf77156c0a49cd08ae8
Reviewed-on: http://gerrit.cloudera.org:8080/18805
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
---
be/CMakeLists.txt | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 953ee9331..4ed427037 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -51,6 +51,12 @@ SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFI
SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_SYSTEM_NO_DEPRECATED")
SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_BIND_GLOBAL_PLACEHOLDERS")
SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_ALLOW_DEPRECATED_HEADERS")
+# -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX
+# For higher portability of the built binaries, switch to /dev/[u]random
+# even if getrandom(2) is available. This is to allow for running binaries
+# built at OS where getrandom(2) is available at OSes where getrandom(2)
+# isn't supported (e.g., that might happen in containerized deployments).
+SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX")
SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -B $ENV{IMPALA_TOOLCHAIN_PACKAGES_HOME}/binutils-$ENV{IMPALA_BINUTILS_VERSION}/bin/")
IF($ENV{USE_GOLD_LINKER} STREQUAL "true")
SET(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -fuse-ld=gold")
[impala] 03/05: IMPALA-11543: Print exception message even if throwable_to_stack_trace_id fails
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit c1bf0af35bf3314c305711dde778038f6df55a82
Author: Csaba Ringhofer <cs...@cloudera.com>
AuthorDate: Tue Aug 30 17:43:32 2022 +0200
IMPALA-11543: Print exception message even if throwable_to_stack_trace_id fails
Until now if we got the message but not the callstack, we did
not print the message, only a generic one about suspecting OOM.
Testing:
- tested manually by throwing an exception from throwableToStackTrace
Change-Id: Ia7d08abb685e687170fc4f3d7a0e38e6c5cae09c
Reviewed-on: http://gerrit.cloudera.org:8080/18930
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/util/jni-util.cc | 2 ++
1 file changed, 2 insertions(+)
diff --git a/be/src/util/jni-util.cc b/be/src/util/jni-util.cc
index 8792f7863..31dcf6399 100644
--- a/be/src/util/jni-util.cc
+++ b/be/src/util/jni-util.cc
@@ -279,6 +279,8 @@ Status JniUtil::GetJniExceptionMsg(JNIEnv* env, bool log_stack, const string& pr
env->ExceptionClear();
string oom_msg = Substitute(oom_msg_template, "throwableToStackTrace");
LOG(ERROR) << oom_msg;
+ DCHECK(msg_str_guard.get() != nullptr);
+ LOG(ERROR) << msg_str_guard.get();
return Status(oom_msg);
}
JniUtfCharGuard c_stack_guard;