[jira] [Created] (HUDI-8001) Insert overwrite failed due to missing 'path' property when using Spark 3.5.1 and Hudi 1.0.0

["Ma Jian (Jira)" <ji...@apache.org>]

Ma Jian created HUDI-8001:
-----------------------------

             Summary: Insert overwrite failed due to missing 'path' property when using Spark 3.5.1 and Hudi 1.0.0
                 Key: HUDI-8001
                 URL: https://issues.apache.org/jira/browse/HUDI-8001
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Ma Jian


The issue with Spark 3.5.1 arises because the {{InsertIntoHoodieTableCommand}} chain calls the initialization of the {{HoodieFileIndex}} class. For v1 tables, the path is stored in {{{}CatalogTable#CatalogStorageFormat#storageProperties{}}}, but not in {{{}CatalogTable#properties{}}}. When Spark reloads the table, it removes the path key from {{{}CatalogTable#CatalogStorageFormat#storageProperties{}}}. Consequently, {{InsertIntoHoodieTableCommand}} in Hudi cannot retrieve the path from either {{CatalogTable#CatalogStorageFormat#storageProperties}} or {{CatalogTable#properties}} during {{{}deduceOverwriteConfig{}}}. This absence of the path key in {{combinedOpts}} leads to an error when initializing {{{}HoodieFileIndex{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)