You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/05 19:32:11 UTC
[PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
dongjoon-hyun opened a new pull request, #59241:
URL: https://github.com/apache/spark/pull/59241
### What changes were proposed in this pull request?
This PR aims to fix `ToStringSQLBuilder` to render a `GeneralScalarExpression` with an unknown name as a function call, e.g., `DATE_TRUNC('MONTH', ts)`.
### Why are the changes needed?
Currently, `toString` and `describe` of a V2 expression with an unknown name fail with `StackOverflowError`, because `V2ExpressionSQLBuilder.visitUnexpectedExpr` builds its error message with `toString` of the same expression.
```scala
import org.apache.spark.sql.connector.expressions.{Expression, Expressions, GeneralScalarExpression}
val e = new GeneralScalarExpression("DATE_TRUNC",
Array[Expression](Expressions.literal("MONTH"), Expressions.column("ts")))
e.toString // java.lang.StackOverflowError
```
This is a regression since Apache Spark [v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) by [SPARK-39139](https://issues.apache.org/jira/browse/SPARK-39139).
### Does this PR introduce _any_ user-facing change?
Yes. `toString` and `describe` return a string like `DATE_TRUNC('MONTH', ts)` instead of throwing `StackOverflowError`. Known names are rendered as before, and the SQL builders for pushdown still reject such expressions with `UNEXPECTED_V2_EXPRESSION`.
### How was this patch tested?
Pass the CIs with the newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59241:
URL: https://github.com/apache/spark/pull/59241#issuecomment-6001620974
Could you review this PR, @viirya ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59241:
URL: https://github.com/apache/spark/pull/59241#issuecomment-6006657256
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/5926a10389e8eace9f655dc138f4bb30254f7bed
- merged into branch-4.x https://github.com/apache/spark/commit/0eb25a7e18b2bbbe8d58cb5d96d64df67a8ef40c
- merged into branch-4.3 https://github.com/apache/spark/commit/f7b621b0447170691a3c079f99b58ceb1e50e11a
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59241:
URL: https://github.com/apache/spark/pull/59241#issuecomment-6006901740
This landed
- branch-4.2 via 6b99745a0a5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59241: [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names
URL: https://github.com/apache/spark/pull/59241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59241:
URL: https://github.com/apache/spark/pull/59241#issuecomment-6003528613
Thank you for the review and the detailed verification, @viirya. I added both test cases: a nested case, `ABS(DATE_TRUNC('MONTH', a))`, and an unknown `Predicate` name, `LIKE(a, 'x%')`.
For the question, I agree that `toString` should not throw because it is used in error messages and logs. Since it changes the existing behavior beyond this regression, I'll handle it in a new JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59983][SQL] Fix `StackOverflowError` in `toString` of V2 expressions with unknown names [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59241:
URL: https://github.com/apache/spark/pull/59241#issuecomment-6005662589
Thank you, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org