[jira] [Created] (IMPALA-11484) Create SCAN plan for Iceberg V2 position delete tables

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-11484:
------------------------------------------

             Summary: Create SCAN plan for Iceberg V2 position delete tables
                 Key: IMPALA-11484
                 URL: https://issues.apache.org/jira/browse/IMPALA-11484
             Project: IMPALA
          Issue Type: New Feature
          Components: Frontend
            Reporter: Zoltán Borók-Nagy


Iceberg position delete files store the full URIs and and file positions of rows that are deleted. Therefore we can do an ANTI HASH JOIN between data files and delete files to retrieve only the active rows.

For the data file rows we need to get the virtual columns INPUT_FILE_NAME and FILE_POSITION, while in the delete files we need to retrieve the columns 'file_path' and 'pos': https://iceberg.apache.org/spec/#position-delete-files

Since the data files are in table schema, and the delete files are in a different schema, we need to create a virtual table for the delete files with the corresponding schema.

Iceberg tells us which delete files must be applied to which data files, i.e. if a data file doesn't have a corresponding delete file, the content can be just UNION'ed with the output of the ANTI HASH JOIN.

See more information in the design doc: https://docs.google.com/document/d/1WF_UOanQ61RUuQlM4LaiRWI0YXpPKZ2VEJ8gyJdDyoY/edit#heading=h.5gc49pcc2543



--
This message was sent by Atlassian Jira
(v8.20.10#820010)