You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/04/10 00:57:30 UTC
[PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
yadavay-amzn opened a new pull request, #55285:
URL: https://github.com/apache/spark/pull/55285
### What changes were proposed in this pull request?
Fix the `STRUCT<>` parsing bug by adding a parser-level action to decrement `complex_type_level_counter` when `STRUCT NEQ` is matched in the `dataType` rule.
**Root cause:** When the lexer sees `STRUCT<>`, it tokenizes it as `STRUCT` + `NEQ` (because `NEQ` is defined before `LT`). This increments the counter for `STRUCT` but never decrements it (no `GT` token), corrupting the counter. Subsequent `>>` then fails to be recognized as shift-right.
**Previous attempt:** PR #51480 was merged then reverted because it modified the `NEQ` lexer rule, which broke `ARRAY(col1 <> col2)` where `<>` is the not-equal operator.
**This fix:** Follows @cloud-fan's suggestion to handle it at the parser level. When the parser matches `STRUCT NEQ` in the `dataType` rule, we know `NEQ` is being used as empty angle brackets (not as a comparison), so we decrement the counter via an inline action.
### Why are the changes needed?
`SELECT CAST(null AS STRUCT<>), 2 >> 1` fails with a syntax error because the corrupted counter prevents `>>` from being recognized as shift-right.
### Does this PR introduce _any_ user-facing change?
No. This is a correctness fix for SQL parsing.
### How was this patch tested?
Added test in `SparkSqlParserSuite` covering 6 cases:
1. `STRUCT<>` followed by `>>` (the original bug)
2. Multiple `STRUCT<>` followed by `>>`
3. `STRUCT<>` followed by `>>>` (unsigned shift right)
4. `ARRAY(1 <> 2)` still works (the regression case from the reverted PR)
5. Nested complex types `MAP<STRING, ARRAY<INT>>` still work
6. Mix of empty struct + nested types + shift right
**Test results:**
- `SparkSqlParserSuite`: 46/46 passed
- `DataTypeParserSuite`: 59/59 passed
- `PlanParserSuite`: 80/80 passed
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #55285:
URL: https://github.com/apache/spark/pull/55285#issuecomment-4227554929
@cloud-fan Thanks for the review!
I just requested a JIRA account at https://selfserve.apache.org/jira-account.html
Will comment on the ticket once my request for the account is approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #55285:
URL: https://github.com/apache/spark/pull/55285#issuecomment-4219478300
Hi @yadavay-amzn, thank you for your contribution!
GA workflow seems failed. Could you enable it?
https://github.com/apache/spark/pull/55285/checks?check_run_id=70712131544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55285:
URL: https://github.com/apache/spark/pull/55285#issuecomment-4222682794
@yadavay-amzn please reply in the JIRA ticket, so that I can assign it to you, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55285: [SPARK-52709][SQL] Fix parsing of STRUCT<>
URL: https://github.com/apache/spark/pull/55285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52709][SQL] Fix parsing of STRUCT<> [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #55285:
URL: https://github.com/apache/spark/pull/55285#issuecomment-4220750129
The GA failure seems not related to this change. Could you rebase to `master` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org