You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@gluten.apache.org by "rdtr (via GitHub)" <gi...@apache.org> on 2026/10/05 20:21:26 UTC

[PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

rdtr opened a new pull request, #13206:
URL: https://github.com/apache/gluten/pull/13206

## What changes are proposed in this pull request?

I ran the Spark 4.2 test "SPARK-57725: resolve columns when the input plan has a null-named attribute" in `GlutenDataFrameSuite`, and it failed with a `NullPointerException`:

```
java.lang.NullPointerException: Cannot invoke "String.toLowerCase(java.util.Locale)" because "name" is null
at org.apache.gluten.expression.ConverterUtils$.normalizeColName(ConverterUtils.scala:53)
at org.apache.gluten.expression.ConverterUtils$.getShortAttributeName(ConverterUtils.scala:57)
at org.apache.gluten.expression.ConverterUtils$.genColumnNameWithExprId(ConverterUtils.scala:71)
at org.apache.gluten.expression.ConverterUtils$.collectAttributeNamesWithExprId(ConverterUtils.scala:91)
at org.apache.gluten.substrait.rel.RelBuilder.makeReadRelForInputIterator(RelBuilder.java:163)
```

A column can have a null name. For example, you can make one with a `StructField` that has a null name in the DataFrame API. Spark itself also failed with an NPE on this, and Spark 4.2.0 fixed it (SPARK-57725). The new Spark test selects column `b` from a `LocalRelation` that also has a column with a null name.

In Gluten, the plan is `LocalTableScan [null#0, b#1]` -> `RowToVeloxColumnar` -> `InputIteratorTransformer`. Gluten makes a name like `b#1` (name + `#` + expression ID) for each input column. `normalizeColName` makes the name lower case, but it doesn't check for null, so it throws the NPE.

I didn't check why this bug (looks like not new) is surfaced now, likely because before Spark failed first. Spark 4.2 fixed its side, so now the plan gets to Gluten and the NPE shows up there.

This PR changes `normalizeColName` to return an empty name for a null name. The generated name still has the expression ID, so the column becomes `#0`, and it is still unique. Like in Spark's fix, you can't refer to this column by name, but the query works.

Part of #13179

## How was this patch tested?

- Added a test to `ConverterUtilsSuite`. It calls `genColumnNameWithExprId` and `genColumnNameWithoutExprId` with a column that has a null name, and checks that they return `#<exprId>` and an empty name. It runs on all Spark versions.
- Added a test to `VeloxLocalTableScanSuite`. It does the same thing as the Spark test: it selects `b` from a `LocalRelation` that has a column with a null name. It checks the result and that the Project runs in Velox. It runs with `spark.gluten.sql.columnar.localTableScan` on and off, so it covers both the native LocalTableScan and the `RowToVeloxColumnar` path. It runs only on Spark 4.2 or later, because older Spark versions fail on this plan by themselves.

Results:
- Without this change: the `VeloxLocalTableScanSuite` test fails on Spark 4.2, and the `ConverterUtilsSuite` test fails on Spark 4.1. Both fail with the NPE above.
- With this change, Spark 4.2: `ConverterUtilsSuite` (3 tests) and `VeloxLocalTableScanSuite` (19 tests) pass.
- With this change, Spark 4.1: `ConverterUtilsSuite` (3 tests) and the other 18 tests in `VeloxLocalTableScanSuite` pass.

## Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #13206:
URL: https://github.com/apache/gluten/pull/13206#issuecomment-6003414000

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

Posted by "philo-he (via GitHub)" <gi...@apache.org>.

philo-he commented on PR #13206:
URL: https://github.com/apache/gluten/pull/13206#issuecomment-6094211196

   Updated on GitHub. Will merge the PR soon. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

Posted by "philo-he (via GitHub)" <gi...@apache.org>.

philo-he merged PR #13206:
URL: https://github.com/apache/gluten/pull/13206


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.

zhouyuan commented on PR #13206:
URL: https://github.com/apache/gluten/pull/13206#issuecomment-6083186614

   @rdtr would you please do a rebase? The CI issue should be gone with retry


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-13179][CORE] Fix NPE when a column has a null name [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #13206:
URL: https://github.com/apache/gluten/pull/13206#issuecomment-6091523465

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org