[GitHub] [spark] ryan-johnson-databricks commented on a diff in pull request #40677: [SPARK-43039][SQL] Support custom fields in the file source _metadata column.

["ryan-johnson-databricks (via GitHub)" <gi...@apache.org>]

ryan-johnson-databricks commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1161702693


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +164,17 @@ trait FileFormat {
     }
   }
 
+  /**
+   * Create a file metadata struct column containing fields supported by the given file format.
+   */
+  def createFileMetadataCol(): AttributeReference = {

Review Comment:
   This PR intentionally moved the method _out_ of `FileFormat`, as part of the effort to eliminate the the central code bottleneck antipattern of dumping everything in the `FileFormat` companion object -- which currently makes it impossible for file formats to customize any aspect of metadata handling. Absent a specific reason to forbid file formats from customizing this behavior, I'd prefer to allow it, so we don't have to keep coming back and changing the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org