[jira] [Created] (IMPALA-15241) Use file IDs instead of paths to get INPUT__FILE__NAME

["Csaba Ringhofer (Jira)" <ji...@apache.org>]

Csaba Ringhofer created IMPALA-15241:
----------------------------------------

             Summary: Use file IDs instead of paths to get INPUT__FILE__NAME
                 Key: IMPALA-15241
                 URL: https://issues.apache.org/jira/browse/IMPALA-15241
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend, Frontend
            Reporter: Csaba Ringhofer


See discussion in https://gerrit.cloudera.org/#/c/24572/5//COMMIT_MSG@9 for motivation.

Currently INPUT__FILE__NAME is used in two ways:
1. common: anti join key in IcebergDeleteJoinNode
2. rare: any other use case, e.g. return in select list

INPUT__FILE__NAME is a StringValue where the actual string is deduplicated, but may get duplicated when going through exchanges and some other operators. IMPALA-15171
improves this by nulling the string if it is no longer used after the delete node, but I consider this a bit hacky.

A longer term solution could be to have a global id->path map and store only an int32 in tuples and use it to look up the name if it is actually needed. A similar global map already exists in for directed distribution mode of delete nodes.

The same lookup mechanism could be used for other things too, for example storing partition columns.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)