[jira] [Resolved] (SPARK-42918) Generalize handling of metadata attributes in FileSourceStrategy

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-42918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-42918.
---------------------------------
    Fix Version/s: 3.5.0
       Resolution: Fixed

Issue resolved by pull request 40545
[https://github.com/apache/spark/pull/40545]

> Generalize handling of metadata attributes in FileSourceStrategy
> ----------------------------------------------------------------
>
>                 Key: SPARK-42918
>                 URL: https://issues.apache.org/jira/browse/SPARK-42918
>             Project: Spark
>          Issue Type: Improvement
>          Components: Optimizer
>    Affects Versions: 3.4.1
>            Reporter: Johan Lasperas
>            Priority: Minor
>             Fix For: 3.5.0
>
>
> A first step towards allowing file format implementations to inject custom metadata fields into plans is to make the handling of metadata attributes in `FileSourceStrategy` more generic.
> Today in `FileSourceStrategy` , the lists of constant and generated metadata fields are created manually, checking for known generated fields on one hand and considering the remaining fields as constant metadata fields. We need instead to introduce a way of declaring metadata fields as generated or constant directly in `FileFormat` and propagate that information to `FileSourceStrategy`.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org