You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2021/12/21 14:51:00 UTC

[jira] [Commented] (SPARK-37273) Hidden File Metadata Support for Spark SQL

    [ https://issues.apache.org/jira/browse/SPARK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463291#comment-17463291 ] 

Wenchen Fan commented on SPARK-37273:
-------------------------------------

[~hyukjin.kwon] this is for file source, which still uses the internal FileFormat API by default and the v2 migration is still far away to be completed.

> Hidden File Metadata Support for Spark SQL
> ------------------------------------------
>
>                 Key: SPARK-37273
>                 URL: https://issues.apache.org/jira/browse/SPARK-37273
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.2.0
>            Reporter: Yaohua Zhao
>            Priority: Major
>
> Provide a new interface in Spark SQL that allows users to query the metadata of the input files for all file formats, expose them as *built-in hidden columns* meaning *users can only see them when they explicitly reference them* (e.g. file path, file name)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org