You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "marcuslin123 (via GitHub)" <gi...@apache.org> on 2026/07/30 18:47:46 UTC
[PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
marcuslin123 opened a new pull request, #57656:
URL: https://github.com/apache/spark/pull/57656
### What changes were proposed in this pull request?
This PR fixes several generated-code patterns that Janino accepts but javac rejects:
- Converts `sequence` error message parameters to strings before storing them in `Map<String, String>`.
- Avoids declaring the generated `sequence` array local as `final` before assigning to it.
- Emits primitive `ArrayBuilder` inner class references with source-compatible names such as `ArrayBuilder.ofInt`.
- Uses `Iterator<InternalRow>` for `SortExec`'s generated sorted iterator state, matching the sorter return type.
- Creates mutable state arrays for generic Java types with raw array creation while preserving the generic field type.
It also adds focused regression coverage for these generated-code shapes.
### Why are the changes needed?
Spark currently relies on Janino for whole-stage codegen, and Janino tolerates some constructs that are not valid Java source for javac. These issues block javac-compatible generated-code paths and can also lead to incorrect runtime behavior for `sequence` error parameters because non-`String` values are inserted into a `Map<String, String>`.
### Does this PR introduce _any_ user-facing change?
Yes. For invalid `sequence` boundaries in the codegen path, Spark now stores string message parameters as declared by the thrown `SparkIllegalArgumentException`. The remaining changes are generated Java source compatibility fixes with no intended behavior change.
### How was this patch tested?
Added regression tests and ran:
```
build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.CodeGenerationSuite -- -z "SPARK-58437"' 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.CollectionExpressionsSuite -- -z "Sequence of numbers"' 'sql/testOnly org.apache.spark.sql.execution.WholeStageCodegenSuite -- -z "SPARK-58437"'
```
Also ran:
```
git diff --check
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (OpenAI GPT-5) was used for code assistance and review support; the patch was guided by the contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5146029169
Thank you for your contribution, @marcuslin123 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #57656:
URL: https://github.com/apache/spark/pull/57656#discussion_r3691528533
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:
##########
@@ -402,6 +402,18 @@ class CodegenContext extends Logging {
}
def declareMutableStates(): String = {
+ def rawJavaType(javaType: String): String = {
Review Comment:
`addMutableState`'s scaladoc (line 306 in this file) still says "generic type arguments are accepted but ignored", which stops being accurate with this change - the type arguments are now kept in the field declaration and only stripped from the array creation. Worth updating in the same PR so the doc doesn't contradict `declareMutableStates`, e.g.:
```scala
* Generic type arguments are kept in the field declaration but stripped from
* array creation expressions, since Java forbids generic array creation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57656:
URL: https://github.com/apache/spark/pull/57656#discussion_r3686145647
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala:
##########
@@ -1114,6 +1115,21 @@ class CollectionExpressionsSuite
checkExceptionInExpression[IllegalArgumentException](
new Sequence(Literal(1), Literal(2), Literal(-1)), EmptyRow, "boundaries: 1 to 2 by -1")
Review Comment:
The new String.valueOf fix in genSequenceLengthCode is exercised only for IntegerType. genSequenceLengthCode is shared by IntegralSequenceImpl (int/long primitives) and InternalSequenceBase (timestamp/date sequences, which pass long microsecond values for $start/$stop and a CalendarInterval or long for $step). A test covering a timestamp or date sequence with invalid boundaries in CODEGEN_ONLY mode would confirm the fix propagates to that call path and document what the error message looks like. Not a blocker, but the coverage gap leaves a distinct call path unexercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #57656:
URL: https://github.com/apache/spark/pull/57656#discussion_r3688194740
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala:
##########
@@ -3992,9 +3992,9 @@ object Sequence {
| ($estimatedStep < 0 && $start >= $stop) ||
| ($estimatedStep == 0 && $start == $stop))) {
| java.util.Map<String, String> params = new java.util.HashMap<String, String>();
- | params.put("start", $start);
- | params.put("stop", $stop);
- | params.put("step", $step);
+ | params.put("start", String.valueOf($start));
Review Comment:
I've moved this to SPARK-58440, as it's an independent bug fix that needs to be backported to older versions, while the other changes are preparatory work for SPARK-57403.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5150092056
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/a4c18a8ddf40f8dda7b676487ab17b12206ec501
- merged into branch-4.x https://github.com/apache/spark/commit/fbcf1ff3513ce8d0de377fa8769e5eb2e7228c57
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5145845902
cc @dongjoon-hyun This is one of the preparatory tasks for SPARK-57370.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "marcuslin123 (via GitHub)" <gi...@apache.org>.
marcuslin123 commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5150158682
> Thank you @marcuslin123 @dongjoon-hyun
>
> @marcuslin123 Additionally, what is your Jira ID? I can assign this ticket to you accordingly.
My Jira ID is marcuslin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #57656: [SPARK-58437][SQL] Fix invalid Java constructs in generated code
URL: https://github.com/apache/spark/pull/57656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5145978210
Thank you for pinging me, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58437][SQL] Fix invalid Java constructs in generated code [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #57656:
URL: https://github.com/apache/spark/pull/57656#issuecomment-5150098272
Thank you @marcuslin123 @dongjoon-hyun
@marcuslin123 Additionally, what is your Jira ID? I can assign this ticket to you accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org