You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Rohit Mittapalli (Jira)" <ji...@apache.org> on 2024/01/16 22:54:00 UTC

[jira] [Created] (HUDI-7300) Parquet DFS source should support merging schemas

Rohit Mittapalli created HUDI-7300:
--------------------------------------

             Summary: Parquet DFS source should support merging schemas
                 Key: HUDI-7300
                 URL: https://issues.apache.org/jira/browse/HUDI-7300
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: Rohit Mittapalli


We should surface the option to merge schema across the parquet files in a single commit. when using ParquetDFSSource.

 

When false the schema is randomly picked from a parquet file (current behavior). When set to true the schema across a commit is merged.

 

https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#schema-merging



--
This message was sent by Atlassian Jira
(v8.20.10#820010)