You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhoulii (via GitHub)" <gi...@apache.org> on 2026/08/26 10:29:13 UTC
[I] [spark] Reading a newly added BLOB column fails after MERGE due to unresolved placeholders [paimon]
zhoulii opened a new issue, #9412:
URL: https://github.com/apache/paimon/issues/9412
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
2.0
### Compute Engine
spark3.2
### Minimal reproduce step
CREATE TABLE t (id INT, name STRING)
TBLPROPERTIES (
'row-tracking.enabled' = 'true',
'data-evolution.enabled' = 'true'
);
-- Keep the historical rows in one normal anchor.
INSERT INTO t
SELECT /*+ REPARTITION(1) */ *
FROM VALUES
(1, 'name1'),
(2, 'name2'),
(3, 'name3') AS v(id, name);
-- The historical anchor has no physical baseline for this column.
ALTER TABLE t
ADD COLUMN picture BINARY COMMENT '__BLOB_FIELD';
-- Create a snapshot using the new schema to avoid the unrelated
-- no-new-schema-snapshot planning issue.
INSERT INTO t VALUES (4, 'name4', CAST(NULL AS BINARY));
CREATE TABLE s (id INT, picture BINARY);
INSERT INTO s VALUES (1, X'4E4557');
MERGE INTO t
USING s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.picture = s.picture;
SELECT id, picture
FROM t
WHERE id <= 3
ORDER BY id;
This reproduces with both Spark V1 and V2 write paths.
### What doesn't meet your expectations?
The query should return the updated BLOB for row 1 and NULL for the unchanged historical rows:
1, X'4E4557'
2, NULL
3, NULL
Instead, it fails while converting an internal placeholder to Spark BINARY:
java.lang.UnsupportedOperationException:
Should never call this method for placeholder blob.
at org.apache.paimon.data.BlobPlaceholder.toData
at org.apache.paimon.spark.data.Spark3InternalRowWithBlob.getBinary
The MERGE writes a full-range BLOB file for the historical anchor. The updated row contains the new value, while the unchanged rows contain placeholders.
Because the file has one sequence group and fully covers the logical range, DataEvolutionSplitRead uses the sequentialReadFiles fast path. This bypasses BlobFallbackRecordReader, so unresolved placeholders are returned directly to Spark.
Full physical row-id coverage does not imply that the file is placeholder-free.
### Anything else?
Do not bypass placeholder resolution for BlobFileBunch.
Remove the sequentialReadOptimize() fast path in DataEvolutionSplitRead and route all BLOB bunches through BlobFallbackRecordReader.
BlobFallbackRecordReader already provides the required semantics:
- Return the latest non-placeholder value.
- Fall back to older sequence groups when necessary.
- Return NULL when no physical baseline exists.
- Handle BLOB, ARRAY<BLOB>, and MAP<X, BLOB> consistently.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [spark] Reading a newly added BLOB column fails after MERGE due to unresolved placeholders [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #9412: [spark] Reading a newly added BLOB column fails after MERGE due to unresolved placeholders
URL: https://github.com/apache/paimon/issues/9412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org