You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Stephen0421 (via GitHub)" <gi...@apache.org> on 2026/07/30 08:19:49 UTC
[PR] [core] Support BLOB descriptors in partial-update tables. [paimon]
Stephen0421 opened a new pull request, #8928:
URL: https://github.com/apache/paimon/pull/8928
### Purpose
Support scalar BLOB descriptor fields in primary-key tables using the partial-update merge engine.
- Reject managed BLOB and BLOB view fields for partial-update tables.
- Preserve specific validation errors for BLOB primary-key, bucket-key, and sequence-field configurations.
- Support descriptor fields in sequence groups, including null, equal-sequence, newer-sequence, and older-sequence semantics.
- Verify descriptor preservation through full compaction.
- Document supported configurations and null handling.
### Tests
- `SchemaValidationTest`
- `PartialUpdateMergeFunctionTest`
- `PartialUpdateITCase#testBlobDescriptorPartialUpdate`
- `PartialUpdateITCase#testBlobDescriptorPartialUpdateSequenceGroup`
- `PartialUpdateITCase#testPartialUpdateRejectsBlobViewField`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support BLOB fields in partial-update tables. [paimon]
Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on code in PR #8928:
URL: https://github.com/apache/paimon/pull/8928#discussion_r3705266963
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -1427,6 +1428,36 @@ private static void validatePrimaryKeyBlobConfiguration(
return;
}
+ checkArgument(
+ options.mergeEngine() == MergeEngine.DEDUPLICATE
+ || options.mergeEngine() == MergeEngine.PARTIAL_UPDATE,
Review Comment:
Thanks for catching this. I fixed it at the schema-validation boundary.
For a managed BLOB field protected by a sequence group, retract-sensitive aggregate functions are now rejected because managed BLOB payloads are not retained in retract records. `last_value` remains supported because it can deterministically clear the field without the original payload, and other aggregate functions can be used with `fields.<field>.ignore-retract=true`.
The validation still allows safe cases such as `ignore-delete=true` and managed BLOB fields that are not protected by a sequence group.
I also added end-to-end coverage for newer and older retract sequences, compaction, empty compacted `.blobref` files, and `ignore-retract` with scalar, ARRAY, and MAP managed BLOB fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support BLOB fields in partial-update tables. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8928:
URL: https://github.com/apache/paimon/pull/8928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support BLOB fields in partial-update tables. [paimon]
Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on PR #8928:
URL: https://github.com/apache/paimon/pull/8928#issuecomment-5139817733
> Why reject managed BLOB and BLOB view fields for partial-update tables?
Actually it can support, already added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support BLOB fields in partial-update tables. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8928:
URL: https://github.com/apache/paimon/pull/8928#discussion_r3703503721
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -1427,6 +1428,36 @@ private static void validatePrimaryKeyBlobConfiguration(
return;
}
+ checkArgument(
+ options.mergeEngine() == MergeEngine.DEDUPLICATE
+ || options.mergeEngine() == MergeEngine.PARTIAL_UPDATE,
Review Comment:
Allowing `PARTIAL_UPDATE` here exposes an incompatible retract path for managed BLOBs. `PrimaryKeyBlobExternalizer.externalize` replaces every non-null managed BLOB with `null` for `DELETE`/`UPDATE_BEFORE`, while `PartialUpdateMergeFunction` passes that field to `FieldAggregator.retract`. With `fields.ts.sequence-group=payload` and `fields.payload.aggregate-function=last_non_null_value`, inserting `(payload=A, ts=1)` and then retracting `(payload=A, ts=2)` incorrectly keeps `A`, because a null retract field means ‘keep the accumulator.’ Please either preserve/externalize non-null managed BLOB values on retract (and retain their references), or reject retract-sensitive aggregation for managed BLOB fields. An end-to-end retract regression test would also be needed.
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -1427,6 +1428,36 @@ private static void validatePrimaryKeyBlobConfiguration(
return;
}
+ checkArgument(
+ options.mergeEngine() == MergeEngine.DEDUPLICATE
+ || options.mergeEngine() == MergeEngine.PARTIAL_UPDATE,
+ "Primary-key managed BLOB tables only support the deduplicate or "
+ + "partial-update merge engine.");
+ checkArgument(
+ options.changelogProducer() == ChangelogProducer.NONE,
+ "Primary-key managed BLOB tables only support changelog-producer 'none'.");
+ checkArgument(
+ options.dataFileExternalPaths() == null,
+ "Primary-key managed BLOB tables do not support '%s'.",
+ CoreOptions.DATA_FILE_EXTERNAL_PATHS.key());
+ checkArgument(
+ !options.pkClusteringOverride(),
+ "Primary-key managed BLOB tables do not support '%s'.",
+ CoreOptions.PK_CLUSTERING_OVERRIDE.key());
+ }
+
+ private static void validatePrimaryKeyBlobKeyConfiguration(
+ TableSchema schema, CoreOptions options) {
+ if (schema.primaryKeys().isEmpty()) {
+ return;
+ }
+
+ Set<String> managedBlobFields =
Review Comment:
This validation only checks the global `sequence.field`; it misses sequence-group ordering fields encoded in option keys. For example, `fields.payload.sequence-group=name` is accepted when `payload` is a managed BLOB in a partial-update table. `PartialUpdateMergeFunction.Factory` later creates a comparator for `payload`, and code generation fails with `IllegalArgumentException: Illegal type: BLOB`. Please parse the key portion of every `fields.<key[,key...]>.sequence-group` option and reject managed `BLOB`, `ARRAY<BLOB>`, and `MAP<K, BLOB>` fields there, while continuing to allow BLOBs on the protected-value side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support BLOB fields in partial-update tables. [paimon]
Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on code in PR #8928:
URL: https://github.com/apache/paimon/pull/8928#discussion_r3705269923
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -1427,6 +1428,36 @@ private static void validatePrimaryKeyBlobConfiguration(
return;
}
+ checkArgument(
+ options.mergeEngine() == MergeEngine.DEDUPLICATE
+ || options.mergeEngine() == MergeEngine.PARTIAL_UPDATE,
+ "Primary-key managed BLOB tables only support the deduplicate or "
+ + "partial-update merge engine.");
+ checkArgument(
+ options.changelogProducer() == ChangelogProducer.NONE,
+ "Primary-key managed BLOB tables only support changelog-producer 'none'.");
+ checkArgument(
+ options.dataFileExternalPaths() == null,
+ "Primary-key managed BLOB tables do not support '%s'.",
+ CoreOptions.DATA_FILE_EXTERNAL_PATHS.key());
+ checkArgument(
+ !options.pkClusteringOverride(),
+ "Primary-key managed BLOB tables do not support '%s'.",
+ CoreOptions.PK_CLUSTERING_OVERRIDE.key());
+ }
+
+ private static void validatePrimaryKeyBlobKeyConfiguration(
+ TableSchema schema, CoreOptions options) {
+ if (schema.primaryKeys().isEmpty()) {
+ return;
+ }
+
+ Set<String> managedBlobFields =
Review Comment:
Thanks, fixed. Schema validation now parses the key portion of every partial-update `fields.<key[,key...]>.sequence-group` option and recursively rejects ordering fields containing `BLOB`, including `ARRAY<BLOB>` and `MAP<K, BLOB>`.
BLOB fields on the protected-value side remain supported. The parser is shared with `PartialUpdateMergeFunction.Factory` to keep schema validation and runtime behavior consistent.
I also added tests for scalar, ARRAY, MAP, multi-key ordering fields, allowed BLOB protected fields, malformed sequence-group options, and compatibility with non-partial-update merge engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org