You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/09/26 16:58:53 UTC
[PR] [core] Rebuild the default value row when the write type changes [paimon]
LuciferYang opened a new pull request, #10208:
URL: https://github.com/apache/paimon/pull/10208
### Purpose
`TableWriteImpl` builds its `DefaultValueRow` once in the constructor, sized to the constructor row type. `withWriteType` refreshed the write type and the not-null indexes but left the `DefaultValueRow` untouched, so once the write type changed the stale default value row no longer matched the rows it wraps. The Spark copy-on-write rewrite writes with the row-tracking-extended type, and a partial write uses a narrowed projection, both of which shift arity and positions. Reading an appended position that is null, such as the null sequence number the rewrite carries, fell through to the default row's underlying array of the old length and crashed the write with an `ArrayIndexOutOfBoundsException`. This rebuilds the default value row from the new write type so its arity and positions follow the type.
### Tests
Added `testWithWriteTypeRebuildsDefaultValueRow`: after switching to the row-tracking-extended write type with a null sequence number, an explicit value on the defaulted column survives the type change.
Added `testWithWriteTypeSubstitutesDefaultValue`: same shape but the defaulted column is left null, so the rebuilt default value row fills the column default at its new position.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #10208: [core] Rebuild the default value row when the write type changes
URL: https://github.com/apache/paimon/pull/10208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #10208:
URL: https://github.com/apache/paimon/pull/10208#issuecomment-5996264735
Thank you @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10208:
URL: https://github.com/apache/paimon/pull/10208#issuecomment-5996219677
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #10208:
URL: https://github.com/apache/paimon/pull/10208#issuecomment-5956355998
@JingsongLi Could you take a look this one? Spark UPDATE on a row-tracking table with a column DEFAULT fails with `ArrayIndexOutOfBoundsException`; both tests are end-to-end (Spark SQL and a Flink action).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10208:
URL: https://github.com/apache/paimon/pull/10208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rebuild the default value row when the write type changes [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10208:
URL: https://github.com/apache/paimon/pull/10208#discussion_r4178165866
##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableWriteImpl.java:
##########
@@ -264,6 +268,10 @@ private void checkNullability(InternalRow row, RowKind rowKind) {
}
private InternalRow wrapDefaultValue(InternalRow row) {
+ if (defaultValueRowOutdated) {
+ defaultValueRow = DefaultValueRow.create(writeType);
Review Comment:
[P2] Preserve ROW defaults when rebuilding for a nested partial write
The Java `BatchTableWrite.withWriteType(...).write(...)` path also wraps rows (as exercised by `NestedSubfieldDataEvolutionTableTest`). For a table with one column `nest ROW<a INT,b STRING> DEFAULT {42,z}`, enabling row tracking/data evolution/nested fields and writing with `table.rowType().projectByPaths(["nest.a"])` retains the full `{42,z}` default in the projected field metadata. This new `create(writeType)` consequently throws `Row field count mismatch. Expected: 1, Actual: 2`, even for an explicitly nonnull `GenericRow.of(GenericRow.of(100))`. I reproduced this with a real FileSystemCatalog: commit `(10,x)`, then update only `nest.a`; this head fails before writing, while disabling only `defaultValueRowOutdated` restores the previous behavior and commits/reads `(100,x)`. Please project/translate the cached defaults consistently with the nested write type (or otherwise retain support for this wrapping path), and add a Java partial-write regression test; bypassing the wrapper
in Spark/Flink partial writers does not cover this API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org