You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/01/26 14:17:11 UTC
[I] [BUG][Velox] ALTER TABLE ADD COLUMN with DEFAULT returns NULL for existing data instead of default value [incubator-gluten]
baibaichen opened a new issue, #11494:
URL: https://github.com/apache/incubator-gluten/issues/11494
### Backend
VL (Velox)
### Bug description
**Expected behavior**: When using `ALTER TABLE ADD COLUMN` with a DEFAULT value, existing rows should return the default value for the newly added column when queried (via EXISTS_DEFAULT lazy backfill mechanism).
**Actual behavior**: Existing rows return `NULL` for the newly added column instead of the configured default value.
This affects the EXISTS_DEFAULT lazy backfill mechanism introduced in Spark 3.4.0 (SPARK-38334). While newly inserted data (after ALTER) correctly uses default values, old data (inserted before ALTER) returns NULL.
### Reproduction Steps
```scala
// Test case from GlutenInsertSuite
testGluten("ALTER ADD COLUMN DEFAULT - old INT data backfill") {
withTable("t") {
sql("create table t(a int) using parquet")
// Insert OLD data BEFORE ALTER
sql("insert into t values(1)")
sql("insert into t values(2)")
// ALTER TABLE ADD COLUMN with DEFAULT
sql("alter table t add column (b int default 99)")
// Query old data
spark.table("t").orderBy("a").show()
}
}
```
**Expected Result**:
```
+---+---+
| a| b|
+---+---+
| 1| 99|
| 2| 99|
+---+---+
```
**Actual Result**:
```
+---+----+
| a| b|
+---+----+
| 1|null|
| 2|null|
+---+----+
```
### Test Failure Output
```
== Results ==
!== Correct Answer - 2 == == Spark Answer - 2 ==
!struct<> struct<a:int,b:int>
![1,99] [1,null]
![2,99] [2,null]
```
All 5 test cases in GlutenInsertSuite fail with the same pattern:
1. `ALTER ADD COLUMN DEFAULT - old INT data backfill` - returns NULL instead of 99
2. `ALTER ADD COLUMN DEFAULT - old STRING data backfill` - returns NULL instead of 'unknown'
3. `ALTER ADD COLUMN DEFAULT - old BOOLEAN data backfill` - returns NULL instead of true
4. `ALTER ADD COLUMN DEFAULT - old DATE data backfill` - returns NULL instead of date '2023-01-01'
5. `ALTER ADD COLUMN DEFAULT - multiple types old data backfill` - returns NULL for all default columns
### Gluten version
main branch
### Spark version
Spark-3.4.x, Spark-3.5.x, Spark-4.0.x, Spark-4.1.x
### System information
- Backend: Velox
- File format: Parquet
- Test suite: `gluten-ut/spark41/src/test/scala/org/apache/spark/sql/sources/GlutenInsertSuite.scala`
### Root Cause Analysis
Spark's DEFAULT column implementation uses two metadata keys:
1. **CURRENT_DEFAULT**: Original SQL expression (used for INSERT/UPDATE/MERGE)
2. **EXISTS_DEFAULT**: Constant-folded result (used for backfilling missing columns in existing data)
When reading old Parquet files that don't have the new column, data sources should call `getExistenceDefaultValues()` to fill missing columns with default values. This appears to not be implemented in Velox backend.
### Related Spark Issues
- SPARK-38334: Implement support for DEFAULT values for columns in tables (Spark 3.4.0)
- SPARK-38811: Support for default column values in ALTER TABLE ADD COLUMN
- SPARK-39265: File source v2: support default values for Parquet
### Additional Context
The newly added tests in GlutenInsertSuite (lines 601-721) specifically test the EXISTS_DEFAULT backfill scenario that Spark's own test suite doesn't cover. These tests expose this Velox limitation.
Note: NEW data inserted AFTER the ALTER statement correctly uses default values. Only OLD data (inserted before ALTER) is affected.
---
*This issue was generated with AI assistance (Claude Code).*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] ALTER TABLE ADD COLUMN with DEFAULT returns NULL for existing data instead of default value [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on issue #11494:
URL: https://github.com/apache/incubator-gluten/issues/11494#issuecomment-3809061524
@acvictor let's discuss this issue here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] ALTER TABLE ADD COLUMN with DEFAULT returns NULL for existing data instead of default value [incubator-gluten]
Posted by "acvictor (via GitHub)" <gi...@apache.org>.
acvictor commented on issue #11494:
URL: https://github.com/apache/incubator-gluten/issues/11494#issuecomment-3875668931
@baibaichen I have thought of a fix for Delta/parquet with Velox backend where we can pass EXISTS_DEFAULT metadata from Spark in Gluten through the Substrait layer to Velox and handle it in Velox's Parquet reader.
Spark's reader handles EXISTS_DEFAULT at the file-read level:
- When opening a Parquet file, it compares the physical file schema against the requested read schema
- Columns in the read schema but missing from the file are tracked in missingColumns
- For missing columns with EXISTS_DEFAULT metadata, it fills the column vector with the default value instead of NULLs
- Gluten/Velox bypasses this entirely because it doesn't receive the EXISTS_DEFAULT metadata
I don't believe it is possible to have a single fix that will cover all file formats. Spark itself handles this in a format-specific manner -
- Parquet - VectorizedParquetRecordReader fills missing columns with defaults at file-read time
- ORC - OrcColumnarBatchReader uses ResolveDefaultColumns.existenceDefaultValues() for missing columns
- CSV - UnivocityParser applies defaults using ResolveDefaultColumns.existenceDefaultValues(requiredSchema) on parse error
- JSON - JacksonParser.applyExistenceDefaultValuesToRow() applies defaults after parsing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org