You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gengliangwang (via GitHub)" <gi...@apache.org> on 2026/05/31 00:05:53 UTC
[PR] [SPARK-57180][SQL] Skip statically-dead setNullAt branch in GenerateSafeProjection for non-nullable fields [spark]
gengliangwang opened a new pull request, #56231:
URL: https://github.com/apache/spark/pull/56231
### What changes were proposed in this pull request?
`GenerateSafeProjection` emits `if (isNull) { mutableRow.setNullAt(i); } else { convert; setColumn; }` for every projected field. When the field expression is statically non-nullable, `isNull` is `FalseLiteral`, so the `setNullAt` branch is dead and only the `else` ever runs. This patch detects that case (`evaluationCode.isNull == FalseLiteral` -- the idiom already used by `GenerateUnsafeProjection`) and emits just the conversion + `setColumn`. The nullable path is unchanged. (`GenerateMutableProjection` already branches on `e.nullable`; this brings `GenerateSafeProjection` in line.)
### Why are the changes needed?
Part of SPARK-56908 (umbrella). Removes a dead branch (and the `setNullAt` call) per non-nullable field from the generated safe projection, shrinking the emitted Java for wide non-nullable schemas (helping with the JVM 64KB method / constant-pool limits, Janino compile time, and JIT work).
### Does this PR introduce _any_ user-facing change?
No. The compiled behavior is identical; only the emitted Java source text changes.
### How was this patch tested?
Added `SPARK-57180: SafeProjection over statically non-nullable fields` to `GeneratedProjectionSuite`, projecting non-nullable int/string/struct/array fields through an unsafe -> safe round trip and asserting the values.
```
build/sbt "catalyst/testOnly *GeneratedProjectionSuite" # 10/10
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57180][SQL] Skip statically-dead setNullAt branch in GenerateSafeProjection for non-nullable fields [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #56231: [SPARK-57180][SQL] Skip statically-dead setNullAt branch in GenerateSafeProjection for non-nullable fields
URL: https://github.com/apache/spark/pull/56231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57180][SQL] Skip statically-dead setNullAt branch in GenerateSafeProjection for non-nullable fields [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #56231:
URL: https://github.com/apache/spark/pull/56231#issuecomment-4606114898
@LuciferYang @dongjoon-hyun Thanks for the review. Merging to master/4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org