[jira] [Created] (SPARK-41791) Create distinct metadata attributes for metadata that is constant or file and metadata that is generated during the scan

["Jan-Ole Sasse (Jira)" <ji...@apache.org>]

Jan-Ole Sasse created SPARK-41791:
-------------------------------------

             Summary: Create distinct metadata attributes for metadata that is constant or file and metadata that is generated during the scan
                 Key: SPARK-41791
                 URL: https://issues.apache.org/jira/browse/SPARK-41791
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer
    Affects Versions: 3.3.1
            Reporter: Jan-Ole Sasse
             Fix For: 3.4.0


There are two types or Metadata in Spark
 * Metadata that is constant per file (file_name, file_size, ...)
 * Metadata that is not contant (currently only row_index)

The two types are generated differently
 * File constant metadata is appended to the output after scan
 * non-constant metadata is generated during the scan

The proposal here is to create different metadata attributes to distinguish those different types throughout the code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org