[I] Support INSERT SQL statement with a subset of columns in Spark 3.4 [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #16805:
URL: https://github.com/apache/hudi/issues/16805

   The new tests `TestInsertTable`.`Test Insert Into with subset of columns`, `Test Insert Into with subset of columns on Parquet table` fail on Spark 3.4 due to our validation introduced in HoodieSpark34CatalystPlanUtils in [https://github.com/apache/hudi/pull/11568].  Without this change, INSERT INTO with subset of columns used to work.
   
    
   {code:java}
   override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
     plan match {
       case insert: InsertIntoStatement =>
         // https://github.com/apache/spark/pull/36077
         // first: in this pr, spark34 support default value for insert into, it will regenerate the user specified cols
         //        so, no need deal with it in hudi side
         // second: in this pr, it will append hoodie meta field with default value, has some bug, it look like be fixed
         //         in spark35(https://github.com/apache/spark/pull/41262), so if user want specified cols, need disable default feature.
         if (SQLConf.get.enableDefaultColumns) {
           if (insert.userSpecifiedCols.nonEmpty) {
             throw new AnalysisException("hudi not support specified cols when enable default columns, " +
               "please disable 'spark.sql.defaultColumn.enabled'")
           }
           Some((insert.table, Seq.empty, insert.partitionSpec, insert.query, insert.overwrite, insert.ifPartitionNotExists))
         } else {
           Some((insert.table, insert.userSpecifiedCols, insert.partitionSpec, insert.query, insert.overwrite, insert.ifPartitionNotExists))
         }
       case _ =>
         None
     }
   } {code}
    
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8911
   - Type: Bug
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support INSERT SQL statement with a subset of columns in Spark 3.4 [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #16805:
URL: https://github.com/apache/hudi/issues/16805#issuecomment-5391413165

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8911).
   
   **Findings: still open. The linked PR is merged, but it is not the fix.**
   
   The PR(s) linked to this issue are merged, but each carries a **different JIRA key** from this ticket - they are the change this ticket was filed to follow up on:
   
   - #11568 - `[HUDI-7949] insert into hudi table with columns specified` (HUDI-7949), merged 2024-07-08
   
   That distinction is worth stating explicitly, because a merged linked PR reads like a resolution. Across this whole batch of 134 issues, 43 linked PRs were checked and **not one merged PR carries its own issue's JIRA key** - so none of these tickets can be closed on that basis.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org