You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/12/10 23:10:27 UTC

[PR] [MINOR] Fix outdated api and libraries [hudi]

linliu-code opened a new pull request, #17556:
URL: https://github.com/apache/hudi/pull/17556

   ### Change Logs
   
   1. Cherry-picked a change to reduce docker disk usage
   2. Fixed the outdated dockerhub repo name
   3. Fixed a Spark outdate api usage
   
   ### Impact
   
   In order to pass the GH CI
   
   ### Risk Level
   
   Low
   
   ### Documentation Update
   
   <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
   
   - The config description must be updated if new configs are added or the default value of the configs are changed.
   - Any new feature or user-facing change requires updating the Hudi website. Please follow the 
     [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639361955

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     }, {
       "hash" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107) 
   * 24633de3b80c0863a549b2160ca691f4b0afef86 UNKNOWN
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706271445

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10751",
       "triggerID" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a9e16228873b51bfec8398f2e024e41afdf77519 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748) 
   * 2c3565f11a97b3993eca9745fde4518270da99de Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10751) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2662241719


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   This fixes some flakiness: In GH CI for java17 or 11 test modules, the child thread may execute without setting the java_home, and use some default java libraries that could be different from the one used in compilation, and cause this error. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2662241719


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   This fixes some flakiness: In GH CI for java17 or 11 test modules, the child thread executes without setting the java_home, and uses the "default" java libraries that could be different from the expected one, and cause this error, like 
   
   ```
   TestMergeIntoTable2:
   *** RUN ABORTED ***
     java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable.copy(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.expressions.Expression, scala.collection.Seq, scala.collection.Seq)'
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:114)
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:58)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
     at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:31)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:111)
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706159271

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 19635e252ffd4632c51faaeb7da20cd180466fd6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744) 
   * a9e16228873b51bfec8398f2e024e41afdf77519 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3692178818

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706093179

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   * 19635e252ffd4632c51faaeb7da20cd180466fd6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744) 
   * a9e16228873b51bfec8398f2e024e41afdf77519 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706066342

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   * 19635e252ffd4632c51faaeb7da20cd180466fd6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639341647

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     }, {
       "hash" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107) 
   * 24633de3b80c0863a549b2160ca691f4b0afef86 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2662241719


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   This fixes some flakiness: In GH CI for java17 or 11 test modules, the child thread executes without setting the java_home, and uses the "default" java libraries that could be different from the expected one, and cause this error, like 
   
   `
   TestMergeIntoTable2:
   *** RUN ABORTED ***
     java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable.copy(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.expressions.Expression, scala.collection.Seq, scala.collection.Seq)'
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:114)
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:58)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
     at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:31)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:111)
   
   `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #17556:
URL: https://github.com/apache/hudi/pull/17556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3692177301

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2662241719


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   This fixes some flakiness: In GH CI for java17 or 11 test modules, the child thread executes without setting the java_home, and uses the "default" java libraries that could be different from the expected one, and cause this error, like 
   
   '''
   TestMergeIntoTable2:
   *** RUN ABORTED ***
     java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable.copy(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.expressions.Expression, scala.collection.Seq, scala.collection.Seq)'
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:114)
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:58)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
     at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:31)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:111)
   
   '''



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639714160

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     }, {
       "hash" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 24633de3b80c0863a549b2160ca691f4b0afef86 UNKNOWN
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706260013

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a9e16228873b51bfec8398f2e024e41afdf77519 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639330616

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639670934

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     }, {
       "hash" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107) 
   * 24633de3b80c0863a549b2160ca691f4b0afef86 UNKNOWN
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706417480

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10751",
       "triggerID" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2c3565f11a97b3993eca9745fde4518270da99de Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10751) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639334501

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3639464602

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107",
       "triggerID" : "452e9c01189b961b4f8f1301f27b1fca84e1b475",
       "triggerType" : "PUSH"
     }, {
       "hash" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "24633de3b80c0863a549b2160ca691f4b0afef86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 452e9c01189b961b4f8f1301f27b1fca84e1b475 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10107) 
   * 24633de3b80c0863a549b2160ca691f4b0afef86 UNKNOWN
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2608936183


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   why there is no tests for this? 
   or how come existing tests are succeeding if not for this change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706269508

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2c3565f11a97b3993eca9745fde4518270da99de",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a9e16228873b51bfec8398f2e024e41afdf77519 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748) 
   * 2c3565f11a97b3993eca9745fde4518270da99de UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #17556:
URL: https://github.com/apache/hudi/pull/17556#discussion_r2662241719


##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala:
##########
@@ -111,7 +111,13 @@ case class HoodieSpark32PlusResolveReferences(spark: SparkSession) extends Rule[
       lazy val analyzer = spark.sessionState.analyzer
       val targetTable = if (targetTableO.resolved) targetTableO else analyzer.execute(targetTableO)
       val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
-      val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
+      val originalMergeInto = mO.asInstanceOf[MergeIntoTable]

Review Comment:
   This fixes some flakiness: In GH CI for java17 or 11 test modules, the child thread executes without setting the java_home, and uses the "default" java libraries that could be different from the expected one, and cause this error as follows.
   
   ```
   TestMergeIntoTable2:
   *** RUN ABORTED ***
     java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable.copy(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan, org.apache.spark.sql.catalyst.expressions.Expression, scala.collection.Seq, scala.collection.Seq)'
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:114)
     at org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$$anonfun$apply$1.applyOrElse(HoodieSpark32PlusAnalysis.scala:58)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
     at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:31)
     at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:111)
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706161677

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748",
       "triggerID" : "a9e16228873b51bfec8398f2e024e41afdf77519",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 19635e252ffd4632c51faaeb7da20cd180466fd6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10744) 
   * a9e16228873b51bfec8398f2e024e41afdf77519 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10748) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Fix outdated api usage and libraries [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17556:
URL: https://github.com/apache/hudi/pull/17556#issuecomment-3706063961

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109",
       "triggerID" : "eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "19635e252ffd4632c51faaeb7da20cd180466fd6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * eb7e5d195700f1beb6d6d5d9895f548b4ea0abd2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10109) 
   * 19635e252ffd4632c51faaeb7da20cd180466fd6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org