You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/04/09 20:23:08 UTC
[PR] [SPARK-56426][SQL] Fix LATERAL VIEW column alias with dot in name [spark]
jiwen624 opened a new pull request, #55282:
URL: https://github.com/apache/spark/pull/55282
### What changes were proposed in this pull request?
In `ResolveGenerate`, generator output column names were extracted via `UnresolvedAttribute.name`, which wraps any name containing a dot in backtick characters (e.g. `"skill.inst"` → `` "`skill.inst`" ``). This string was then passed directly to `makeGeneratorOutput` as the resolved attribute name, embedding literal backtick characters into the column name.
The fix uses `nameParts.head` instead, which returns the raw identifier text without display-formatting.
### Why are the changes needed?
Using a backtick-quoted identifier with a dot as a `LATERAL VIEW` column alias causes an `UNRESOLVED_COLUMN` error at analysis time:
```sql
SELECT id, `skill.inst`
FROM VALUES (1, array('a', 'b')) AS (id, skills)
LATERAL VIEW explode(skills) skills_table AS `skill.inst`
-- ERROR: [UNRESOLVED_COLUMN] `skill.inst` cannot be resolved.
-- Did you mean: `skills_table`.```skill.inst```
```
The triple-backtick suggestion reveals that the resolved attribute name contains literal backtick characters, caused by `UnresolvedAttribute.name` being misused as a raw identifier extractor rather than for display only.
This worked in Spark 3.1.x.
### Does this PR introduce _any_ user-facing change?
Yes. Queries using a backtick-quoted `LATERAL VIEW` column alias containing a dot (e.g. `` `a.b` ``) now execute correctly instead of failing with `UNRESOLVED_COLUMN`.
### How was this patch tested?
Added a test case in `SQLQuerySuite` covering `LATERAL VIEW` with a dot-containing column alias.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix LATERAL VIEW column alias with dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55282:
URL: https://github.com/apache/spark/pull/55282#issuecomment-4399810123
gentle ping @cloud-fan @dongjoon-hyun 🙇♂️ for the bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55282: [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name
URL: https://github.com/apache/spark/pull/55282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #55282:
URL: https://github.com/apache/spark/pull/55282#discussion_r3315699407
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3440,6 +3440,17 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
checkAnswer(df2, Row(1) :: Nil)
}
+ test("SPARK-56426: LATERAL VIEW column alias with dot in name should resolve correctly") {
+ checkAnswer(
+ sql(
+ """
+ |SELECT id, `skill.inst`
+ |FROM VALUES (1, array('a', 'b')) AS t(id, skills)
+ |LATERAL VIEW explode(skills) skills_table AS `skill.inst`
Review Comment:
Added 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #55282:
URL: https://github.com/apache/spark/pull/55282#discussion_r3315701256
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3440,6 +3440,17 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
checkAnswer(df2, Row(1) :: Nil)
}
+ test("SPARK-56426: LATERAL VIEW column alias with dot in name should resolve correctly") {
Review Comment:
Moved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix LATERAL VIEW column alias with dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55282:
URL: https://github.com/apache/spark/pull/55282#issuecomment-4277314365
gentle ping @cloud-fan @dongjoon-hyun 🙇♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55282:
URL: https://github.com/apache/spark/pull/55282#discussion_r3312188384
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3190,7 +3190,11 @@ class Analyzer(
throw QueryCompilationErrors.nestedGeneratorError(g.generator)
}
g.copy(generatorOutput =
- GeneratorResolution.makeGeneratorOutput(g.generator, g.generatorOutput.map(_.name)))
+ GeneratorResolution.makeGeneratorOutput(
+ g.generator, g.generatorOutput.map {
+ case ua: UnresolvedAttribute => ua.nameParts.head
Review Comment:
`nameParts.head` (or `.last`) silently drops information if `nameParts` is ever multi-part — and since the resulting name is referenced later in `SELECT`, a silent drop would surface as a confusing `UNRESOLVED_COLUMN` rather than an internal error. In real SQL the parser invariant guarantees a single-part name here (`UnresolvedAttribute.quoted` always wraps in a one-element seq, and the LATERAL VIEW grammar accepts only single identifier tokens for `colName`), but the only non-parser producer of `UnresolvedAttribute` in `generatorOutput` — the test DSL at `dsl/package.scala:524` — uses `UnresolvedAttribute.apply` which *can* yield multi-part names. An `assert` documents the invariant and fails loudly if it's ever violated, instead of silently mis-aliasing.
```suggestion
case ua: UnresolvedAttribute =>
// LATERAL VIEW parser always emits single-part names via
// UnresolvedAttribute.quoted; assert to fail loudly if that ever changes.
assert(ua.nameParts.length == 1, s"unexpected multi-part name: ${ua.nameParts}")
ua.nameParts.head
```
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3440,6 +3440,17 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
checkAnswer(df2, Row(1) :: Nil)
}
+ test("SPARK-56426: LATERAL VIEW column alias with dot in name should resolve correctly") {
Review Comment:
Consider placing this test in `GeneratorFunctionSuite` (the dedicated suite for generator features including LATERAL VIEW — see SPARK-14986, SPARK-37947). `SQLQuerySuite` works, but a feature-specific suite is preferred per the project convention.
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3440,6 +3440,17 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
checkAnswer(df2, Row(1) :: Nil)
}
+ test("SPARK-56426: LATERAL VIEW column alias with dot in name should resolve correctly") {
+ checkAnswer(
+ sql(
+ """
+ |SELECT id, `skill.inst`
+ |FROM VALUES (1, array('a', 'b')) AS t(id, skills)
+ |LATERAL VIEW explode(skills) skills_table AS `skill.inst`
Review Comment:
Optional: add a multi-alias case using a multi-output generator (e.g. `inline(...) AS \`a.b\`, \`c.d\``) to exercise the same code path with multiple dotted aliases. The fix applies element-wise to each entry in `generatorOutput`, so multi-alias coverage broadens regression protection at no extra cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix LATERAL VIEW column alias with dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55282:
URL: https://github.com/apache/spark/pull/55282#issuecomment-4218725947
@cloud-fan @dongjoon-hyun could you take a look at this bug fix when you get a chance? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55282:
URL: https://github.com/apache/spark/pull/55282#issuecomment-4549405938
Hi @cloud-fan @dongjoon-hyun or somebody else: we see this bug in our production env which fails the SQL query. Could you take a look at this fix when you get a chance? 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56426][SQL] Fix SQL failure when LATERAL VIEW column alias contains dot in name [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55282:
URL: https://github.com/apache/spark/pull/55282#issuecomment-4563569491
thanks, merging to master/4.x/4.2 (bug fix)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org