[jira] [Created] (IMPALA-15483) DROP PARTITION leaves delete files in the Iceberg table

["Noémi Pap-Takács (Jira)" <ji...@apache.org>]

Noémi Pap-Takács created IMPALA-15483:
-----------------------------------------

             Summary: DROP PARTITION leaves delete files in the Iceberg table
                 Key: IMPALA-15483
                 URL: https://issues.apache.org/jira/browse/IMPALA-15483
             Project: IMPALA
          Issue Type: Bug
            Reporter: Noémi Pap-Takács


ALTER TABLE <t> DROP PARTITION (<pred>) removes the partition's data files from the table metadata but keeps its delete files — position deletes, equality deletes and deletion vectors — referenced by the new snapshot.

Repro (Iceberg 1.11.0):
{code:java}
CREATE TABLE t (i INT, s STRING) PARTITIONED BY SPEC(i)
STORED BY ICEBERG TBLPROPERTIES ('format-version'='2');
INSERT INTO t VALUES (1,'a'),(1,'b');
INSERT INTO t VALUES (2,'c');
DELETE FROM t WHERE s = 'a';            -- position delete in partition i=1
ALTER TABLE t DROP PARTITION (i = 1);
SELECT content, `partition`.i, file_path FROM t.`files`;
Actual: a content=1, i=1 row remains (.../data/i=1/delete-*.parq).
Expected: no files of partition i=1.{code}

Root cause: AlterTableDropPartitionStmt.analyzeIceberg collects the data files and FileScanTask.deletes() into the path list, and IcebergCatalogOpExecutor.alterTableDropPartition feeds each path to DeleteFiles.deleteFile(String). In Iceberg, MergingSnapshotProducer.delete(CharSequence) registers the path only with the data-file ManifestFilterManager; delete-file manifests are filtered solely through the typed delete(DeleteFile) overload. The delete-file paths are therefore silently dropped, and nothing reports it because validateFilesExist() is not enabled.

Impact: query results are unaffected — with its data files gone the stale delete file matches nothing, and later inserts carry higher data sequence numbers. However the entries persist in the manifests and in the files/delete_files metadata tables, total-delete-files stays inflated, and the files remain referenced, so orphan-file cleanup cannot collect them. Only partial drops are affected: a drop that selects every file takes the is_truncate path (deleteFromRowFilter(alwaysTrue)), which is clean.

Fix direction: delete typed — re-plan the partition at commit time and call deleteFile(DataFile) / deleteFile(DeleteFile), or split the thrift paths field into data-file and delete-file paths.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org