You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nada-attia (via GitHub)" <gi...@apache.org> on 2026/02/09 20:44:34 UTC
[PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
nada-attia opened a new pull request, #18147:
URL: https://github.com/apache/hudi/pull/18147
### Describe the issue this Pull Request addresses
When Spark SQL queries contain unresolved columns or tables (e.g., typos, missing table definitions), users receive a cryptic error message like "Invalid call to dataType on unresolved object" which provides no actionable information. This PR improves error handling to catch `UnresolvedException` and provide user-friendly error messages that help users identify and fix the issue.
### Summary and Changelog
**Summary:** Improved error handling in Hudi's Spark SQL analysis phase to provide clear, actionable error messages when queries contain unresolved references.
**Changelog:**
- Modified `ProducesHudiMetaFields.unapply` in `HoodieAnalysis.scala` to catch `UnresolvedException`
- Added `collectUnresolvedReferences` helper method to identify specific unresolved column/table names
- Throws `AnalysisException` with helpful error message including:
- List of unresolved references found in the query
- Suggestions to check for typos, missing table definitions, incorrect schema references
- Original error message for debugging
### Impact
- **User-facing change:** Users will now see clear error messages like:
```
Failed to resolve query. The query contains unresolved columns or tables. Unresolved references: [nonexistent_column].
Please check for: (1) typos in column or table names, (2) missing table definitions,
(3) incorrect database/schema references, (4) columns that don't exist in the source tables.
```
- **No API changes**
- **No performance impact**
### Risk Level
**Low** - This change only affects error handling in the analysis phase. It catches a specific exception type (`UnresolvedException`) and re-throws it as a more informative `AnalysisException`. Normal query execution paths are unaffected.
### Documentation Update
None - This is an internal improvement to error messages that doesn't require documentation updates.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3874777709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3898322776
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2186107c3916396ef04db38d99485329f3cf621e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4814810067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
}, {
"hash" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 553550c102465332b98cd1e6a8d828749ee176d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891)
* f7cb1de9f06ca3ffe4561333d84765aa9c478820 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4766501145
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9cfcdf0eb6cb0142c1582c6b68720423c980eab6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4782581945
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9cfcdf0eb6cb0142c1582c6b68720423c980eab6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862)
* 553550c102465332b98cd1e6a8d828749ee176d2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18147:
URL: https://github.com/apache/hudi/pull/18147#discussion_r2795763747
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ Some(resolved.output)
+ } else {
+ None
+ }
+ } catch {
+ case e: UnresolvedException =>
+ val unresolvedRefs = collectUnresolvedReferences(plan)
Review Comment:
This extractor is used inside pattern matches like `query match { case ProducesHudiMetaFields(output) => ...; case _ => None }`. Before this change, an `UnresolvedException` from `analyzer.execute()` would propagate naturally and Spark's own analyzer would eventually produce its own (usually quite good) error message. Now we're catching it and throwing a new `AnalysisException` — this short-circuits Spark's normal error-handling and could surface a less precise Hudi-specific message for cases where Spark would have reported the exact column/table problem. Have you considered returning `None` here instead of throwing, and letting Spark's built-in analysis error reporting handle it? That would preserve the pattern-match fallthrough semantics this extractor is designed for.
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ Some(resolved.output)
+ } else {
+ None
+ }
+ } catch {
+ case e: UnresolvedException =>
+ val unresolvedRefs = collectUnresolvedReferences(plan)
+ val unresolvedInfo = if (unresolvedRefs.nonEmpty) {
+ s" Unresolved references: [${unresolvedRefs.mkString(", ")}]."
+ } else {
+ ""
+ }
+ throw new AnalysisException(
+ s"Failed to resolve query. The query contains unresolved columns or tables.$unresolvedInfo " +
+ s"Please check for: (1) typos in column or table names, (2) missing table definitions, " +
Review Comment:
It would be good to check version-specific exception handling in Hudi Spark integration as well.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/analysis/TestHoodieAnalysisErrorHandling.scala:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase
+
+/**
+ * Tests for improved error handling in HoodieAnalysis when queries contain
+ * unresolved columns or tables.
+ */
+class TestHoodieAnalysisErrorHandling extends HoodieSparkSqlTestBase {
+
+ test("MergeInto with unresolved column in source query should provide helpful error message") {
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ // Create target Hudi table
+ spark.sql(
+ s"""
+ |CREATE TABLE $tableName (
+ | id INT,
+ | name STRING,
+ | price DOUBLE,
+ | ts INT
+ |) USING hudi
+ |LOCATION '${tmp.getCanonicalPath}'
+ |TBLPROPERTIES (
+ | primaryKey = 'id',
+ | preCombineField = 'ts'
+ |)
+ """.stripMargin)
+
+ // Insert initial data
+ spark.sql(s"INSERT INTO $tableName VALUES (1, 'a1', 10.0, 1000)")
+
+ // Test MERGE INTO with non-existent column in source query
+ val exception = intercept[AnalysisException] {
+ spark.sql(
+ s"""
+ |MERGE INTO $tableName AS target
+ |USING (
+ | SELECT 1 AS id, 'updated' AS name, 20.0 AS price, 2000 AS ts, nonexistent_column AS extra
+ |) AS source
+ |ON target.id = source.id
+ |WHEN MATCHED THEN UPDATE SET *
+ |WHEN NOT MATCHED THEN INSERT *
+ """.stripMargin)
+ }
+
+ // Verify the error message contains helpful information
+ val errorMessage = exception.getMessage
Review Comment:
Several test assertions use `||` with `contains("cannot be resolved")` — that would match Spark's existing error messages even without this PR. Could you tighten the assertions to specifically verify the new behavior (e.g., assert on `"Failed to resolve query"` or `"Please check for"`) so these tests actually validate the change? and also account for exception message difference across Spark versions.
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ Some(resolved.output)
+ } else {
+ None
+ }
+ } catch {
+ case e: UnresolvedException =>
+ val unresolvedRefs = collectUnresolvedReferences(plan)
+ val unresolvedInfo = if (unresolvedRefs.nonEmpty) {
+ s" Unresolved references: [${unresolvedRefs.mkString(", ")}]."
+ } else {
+ ""
+ }
+ throw new AnalysisException(
+ s"Failed to resolve query. The query contains unresolved columns or tables.$unresolvedInfo " +
+ s"Please check for: (1) typos in column or table names, (2) missing table definitions, " +
Review Comment:
For example, we have certain fail analysis method and on Spark 3.5 it uses proper classification, which should be used if required:
```
sparkAdapter.getCatalystPlanUtils.failAnalysisForMIT
override def failAnalysisForMIT(a: Attribute, cols: String): Unit = {
a.failAnalysis(
errorClass = "UNRESOLVED_COLUMN.WITH_SUGGESTION",
messageParameters = Map(
"objectName" -> a.sql,
"proposal" -> cols))
}
```
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ Some(resolved.output)
+ } else {
+ None
+ }
+ } catch {
+ case e: UnresolvedException =>
+ val unresolvedRefs = collectUnresolvedReferences(plan)
+ val unresolvedInfo = if (unresolvedRefs.nonEmpty) {
+ s" Unresolved references: [${unresolvedRefs.mkString(", ")}]."
+ } else {
+ ""
+ }
+ throw new AnalysisException(
+ s"Failed to resolve query. The query contains unresolved columns or tables.$unresolvedInfo " +
+ s"Please check for: (1) typos in column or table names, (2) missing table definitions, " +
Review Comment:
For example, on Spark 3.5, the second and fourth test cases already throw readable exception without the changes in this PR:
```
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `target`.`nonexistent_id` cannot be resolved. Did you mean one of the following? [`target`.`name`, `target`.`price`, `target`.`id`, `target`.`ts`, `target`.`_hoodie_file_name`].; line 6 pos 3;
'MergeIntoTable ('target.nonexistent_id = id#106), [updateaction(None, assignment(id#115, id#106), assignment(name#116, name#107), assignment(price#117, price#108), assignment(ts#118, ts#109))], [insertaction(None, assignment(id#115, id#106), assignment(name#116, name#107), assignment(price#117, price#108), assignment(ts#118, ts#109))]
:- SubqueryAlias target
: +- SubqueryAlias spark_catalog.default.htesthoodieanalysiserrorhandling_2
: +- Relation spark_catalog.default.htesthoodieanalysiserrorhandling_2[_hoodie_commit_time#110,_hoodie_commit_seqno#111,_hoodie_record_key#112,_hoodie_partition_path#113,_hoodie_file_name#114,id#115,name#116,price#117,ts#118] HudiFileGroup
+- SubqueryAlias source
+- Project [1 AS id#106, updated AS name#107, 20.0 AS price#108, 2000 AS ts#109]
+- OneRowRelation
at org.apache.spark.sql.errors.QueryCompilationErrors$.unresolvedAttributeError(QueryCompilationErrors.scala:306)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.org$apache$spark$sql$catalyst$analysis$CheckAnalysis$$failUnresolvedAttribute(CheckAnalysis.scala:141)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6(CheckAnalysis.scala:299)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$6$adapted(CheckAnalysis.scala:297)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:244)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)
at scala.collection.Iterator.foreach(Iterator.scala:943)
```
```
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name source.pricee cannot be resolved. Did you mean one of the following? [target._hoodie_commit_seqno, target._hoodie_commit_time, target._hoodie_file_name, target._hoodie_partition_path, target._hoodie_record_key, source.id, target.id, source.name, target.name, source.price, target.price, source.ts, target.ts].; line 10 pos 10
org.apache.spark.sql.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name source.pricee cannot be resolved. Did you mean one of the following? [target._hoodie_commit_seqno, target._hoodie_commit_time, target._hoodie_file_name, target._hoodie_partition_path, target._hoodie_record_key, source.id, target.id, source.name, target.name, source.price, target.price, source.ts, target.ts].; line 10 pos 10
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
at org.apache.spark.sql.HoodieSpark35CatalystPlanUtils$.failAnalysisForMIT(HoodieSpark35CatalystPlanUtils.scala:80)
at org.apache.spark.sql.hudi.analysis.ResolveReferences.$anonfun$resolveMergeExprOrFail$2(HoodieSparkBaseAnalysis.scala:270)
at org.apache.spark.sql.hudi.analysis.ResolveReferences.$anonfun$resolveMergeExprOrFail$2$adapted(HoodieSparkBaseAnalysis.scala:265)
at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:95)
at org.apache.spark.sql.catalyst.expressions.AttributeSet.foreach(AttributeSet.scala:137)
at org.apache.spark.sql.hudi.analysis.ResolveReferences.resolveMergeExprOrFail(HoodieSparkBaseAnalysis.scala:265)
at org.apache.spark.sql.hudi.analysis.ResolveReferences.$anonfun$resolveAssignments$1(HoodieSparkBaseAnalysis.scala:254)
```
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ Some(resolved.output)
+ } else {
+ None
+ }
+ } catch {
+ case e: UnresolvedException =>
+ val unresolvedRefs = collectUnresolvedReferences(plan)
+ val unresolvedInfo = if (unresolvedRefs.nonEmpty) {
+ s" Unresolved references: [${unresolvedRefs.mkString(", ")}]."
+ } else {
+ ""
+ }
+ throw new AnalysisException(
+ s"Failed to resolve query. The query contains unresolved columns or tables.$unresolvedInfo " +
+ s"Please check for: (1) typos in column or table names, (2) missing table definitions, " +
Review Comment:
I wonder if the error message is too Hudi-specific for something that's really a general Spark analysis failure. Spark already provides messages like below — wrapping it in a generic "check for typos" message might actually lose information. Could we revisit if the exception throwing logic to follow Spark standard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3898097588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae7d5f06d91747e8a0c6af88ea474d3e1021bc69 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998)
* 2186107c3916396ef04db38d99485329f3cf621e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4765649660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2186107c3916396ef04db38d99485329f3cf621e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017)
* 9cfcdf0eb6cb0142c1582c6b68720423c980eab6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3887313023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873)
* 42c3adcc0415a9183c7bbcaf18d14fccca963e48 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3898083264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae7d5f06d91747e8a0c6af88ea474d3e1021bc69 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998)
* 2186107c3916396ef04db38d99485329f3cf621e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3875691561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18147:
URL: https://github.com/apache/hudi/pull/18147#discussion_r2796386493
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -298,19 +298,59 @@ object HoodieAnalysis extends SparkAdapterSupport {
private object ProducesHudiMetaFields {
def unapply(plan: LogicalPlan): Option[Seq[Attribute]] = {
- val resolved = if (plan.resolved) {
- plan
- } else {
- val analyzer = spark.sessionState.analyzer
- analyzer.execute(plan)
- }
+ try {
+ val resolved = if (plan.resolved) {
+ plan
+ } else {
+ val analyzer = spark.sessionState.analyzer
+ analyzer.execute(plan)
+ }
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
- Some(resolved.output)
- } else {
- None
+ if (resolved.output.exists(attr => isMetaField(attr.name))) {
Review Comment:
Instead, should we fix these places to let Spark's analysis produce a clear error?
```
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSparkBaseAnalysis.scala
166 val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
167 val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
168 // END: custom Hudi change
169 - EliminateSubqueryAliases(targetTable) match {
169 + // If source table still has unresolved references (e.g., non-existent columns/tables),
170 + // return the partially resolved plan and let Spark's CheckAnalysis produce a clear error.
171 + if (!sourceTable.resolved) {
172 + m
173 + } else EliminateSubqueryAliases(targetTable) match {
174 case r: NamedRelation if r.skipSchemaResolution =>
175 // Do not resolve the expression if the target table accepts any schema.
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
305 analyzer.execute(plan)
306 }
307
308 - if (resolved.output.exists(attr => isMetaField(attr.name))) {
308 + // If the plan is still not resolved (e.g., references non-existent tables/columns),
309 + // skip meta-field checks and let Spark's standard analysis produce a clear error.
310 + if (resolved.resolved && resolved.output.exists(attr => isMetaField(attr.name))) {
311 Some(resolved.output)
312 } else {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3873779288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bcfd40c0e809b4fcdad856fb27fb77eaf61958ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bcfd40c0e809b4fcdad856fb27fb77eaf61958ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bcfd40c0e809b4fcdad856fb27fb77eaf61958ab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4815257445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
}, {
"hash" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14976",
"triggerID" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7cb1de9f06ca3ffe4561333d84765aa9c478820 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892403249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65519ec10e26f3a278d0994b7f6c87c8f83ca88f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4765678566
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2186107c3916396ef04db38d99485329f3cf621e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017)
* 9cfcdf0eb6cb0142c1582c6b68720423c980eab6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #18147:
URL: https://github.com/apache/hudi/pull/18147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892997399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae7d5f06d91747e8a0c6af88ea474d3e1021bc69 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3874759924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892781818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65519ec10e26f3a278d0994b7f6c87c8f83ca88f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991)
* ae7d5f06d91747e8a0c6af88ea474d3e1021bc69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3887323583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873)
* 42c3adcc0415a9183c7bbcaf18d14fccca963e48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892117370
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42c3adcc0415a9183c7bbcaf18d14fccca963e48 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948)
* 65519ec10e26f3a278d0994b7f6c87c8f83ca88f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892101382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42c3adcc0415a9183c7bbcaf18d14fccca963e48 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948)
* 65519ec10e26f3a278d0994b7f6c87c8f83ca88f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3892765889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65519ec10e26f3a278d0994b7f6c87c8f83ca88f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991)
* ae7d5f06d91747e8a0c6af88ea474d3e1021bc69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4814826515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
}, {
"hash" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14976",
"triggerID" : "f7cb1de9f06ca3ffe4561333d84765aa9c478820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 553550c102465332b98cd1e6a8d828749ee176d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891)
* f7cb1de9f06ca3ffe4561333d84765aa9c478820 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-3887712743
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42c3adcc0415a9183c7bbcaf18d14fccca963e48 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4765630557
Hi @nada-attia, picking this up to address @yihua's review.
The original approach — catch `UnresolvedException` in `ProducesHudiMetaFields.unapply` and rewrite it as a generic Hudi error — does close out the cryptic message, but as @yihua [pointed out](https://github.com/apache/hudi/pull/18147#discussion_r2795763747) it intercepts Spark before `CheckAnalysis` runs, so we lose Spark's structured `UNRESOLVED_COLUMN.WITH_SUGGESTION` errors with "did you mean" hints. Reworked to follow @yihua's suggestion in [#discussion_r2796386493](https://github.com/apache/hudi/pull/18147#discussion_r2796386493): let Spark's analyzer surface its own error.
Force-pushed (squashed the 6 commits) — diff is now small (3 files, ~190 lines, mostly the test).
### Code changes
**`HoodieAnalysis.ProducesHudiMetaFields.unapply`** — gate the output inspection on `resolved.resolved`. If the plan is still not fully resolved, return `None` so the pattern-match falls through; Spark's `CheckAnalysis` then runs in its normal phase and produces the precise error.
```diff
- if (resolved.output.exists(attr => isMetaField(attr.name))) {
+ // If the plan is still not fully resolved (e.g., references non-existent
+ // tables or columns), fall through. Spark's CheckAnalysis runs later and
+ // produces precise UNRESOLVED_COLUMN / TABLE_OR_VIEW_NOT_FOUND errors with
+ // "did you mean" suggestions; intercepting UnresolvedException here would
+ // discard that context. Only inspect the output once the plan is resolved.
+ if (resolved.resolved && resolved.output.exists(attr => isMetaField(attr.name))) {
```
**`HoodieSparkBaseAnalysis.ResolveReferences` (MIT case)** — if `sourceTable` is still unresolved after `analyzer.execute`, return the partially-resolved `MergeIntoTable` plan instead of descending into `resolveAssignments`. Spark's downstream rules then handle the error correctly.
```diff
val sourceTable = if (sourceTableO.resolved) sourceTableO else analyzer.execute(sourceTableO)
val m = mO.asInstanceOf[MergeIntoTable].copy(targetTable = targetTable, sourceTable = sourceTable)
- EliminateSubqueryAliases(targetTable) match {
+ if (!sourceTable.resolved) {
+ m // let Spark's CheckAnalysis surface the unresolved-source error
+ } else EliminateSubqueryAliases(targetTable) match {
```
### Reverted from the prior version
- `failUnresolvedQuery` on the `HoodieCatalystPlansUtils` trait and the four version-specific impls (3.3 / 3.4 / 3.5 / 4.0)
- `collectUnresolvedReferences` helper in `HoodieAnalysis`
- The `try/catch UnresolvedException` block in `ProducesHudiMetaFields.unapply`
- The unused `UnresolvedException` import
### Tests
Rewrote `TestHoodieAnalysisErrorHandling`:
- 3 scenarios: MERGE INTO with unresolved column in source query, MERGE INTO with unresolved column in ON predicate, INSERT INTO from non-existent source table.
- Assertions now check that Spark's native error patterns (`UNRESOLVED_COLUMN` / `TABLE_OR_VIEW_NOT_FOUND` / "cannot be resolved") are present **and** that the specific column/table name appears in the message.
- Each test also asserts that the prior Hudi wrapper (`"Failed to resolve query"` / `"Please check for: (1) typos"`) is NOT present, so a future regression that re-introduces the rewrap fails the test.
- Helpers accept both Spark 3.3 (pre-error-class) and Spark 3.4+ (error-class) phrasings, addressing the version-difference concern in [#discussion_r2795763752](https://github.com/apache/hudi/pull/18147#discussion_r2795763752).
### Rebase
Rebased onto latest master (was 424 commits behind).
@yihua @danny0405 PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4782543148
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9cfcdf0eb6cb0142c1582c6b68720423c980eab6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862)
* 553550c102465332b98cd1e6a8d828749ee176d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): preserve Spark's native unresolved-column errors in HoodieAnalysis [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4783229555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11873",
"triggerID" : "1e9a7467c5d0ca2cce291222d0d1f50d0d1ac266",
"triggerType" : "PUSH"
}, {
"hash" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11948",
"triggerID" : "42c3adcc0415a9183c7bbcaf18d14fccca963e48",
"triggerType" : "PUSH"
}, {
"hash" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11991",
"triggerID" : "65519ec10e26f3a278d0994b7f6c87c8f83ca88f",
"triggerType" : "PUSH"
}, {
"hash" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11998",
"triggerID" : "ae7d5f06d91747e8a0c6af88ea474d3e1021bc69",
"triggerType" : "PUSH"
}, {
"hash" : "2186107c3916396ef04db38d99485329f3cf621e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12017",
"triggerID" : "2186107c3916396ef04db38d99485329f3cf621e",
"triggerType" : "PUSH"
}, {
"hash" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14862",
"triggerID" : "9cfcdf0eb6cb0142c1582c6b68720423c980eab6",
"triggerType" : "PUSH"
}, {
"hash" : "553550c102465332b98cd1e6a8d828749ee176d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891",
"triggerID" : "553550c102465332b98cd1e6a8d828749ee176d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 553550c102465332b98cd1e6a8d828749ee176d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14891)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): Improve query error handling for UnresolvedException [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18147:
URL: https://github.com/apache/hudi/pull/18147#issuecomment-4654149055
hey @nada-attia : can you take a look at the feedback from ethan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org