You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/06 15:01:11 UTC
[PR] [core] Add versioned split serializer [paimon]
JingsongLi opened a new pull request, #8482:
URL: https://github.com/apache/paimon/pull/8482
## Summary
Introduce a versioned binary serializer for non-system table `Split` implementations so split metadata can be exchanged by Java and Rust runtimes through a stable protocol.
## Changes
- Add `SplitSerializer` with a `SPLIT_V1` frame, serializer version, and stable type ids for `DataSplit`, `IncrementalSplit`, `IndexedSplit`, `ChainSplit`, `QueryAuthSplit`, and fallback split variants.
- Update `QueryAuthSplit` and `FallbackSplitImpl` to use custom serialization paths instead of Java default object serialization.
- Add round-trip and golden-file compatibility coverage, including generated golden files for all supported split variants.
## Testing
- [x] `mvn -pl paimon-core -Pfast-build -DgenerateSplitGoldenFiles=true -Dtest=SplitSerializerTest,QueryAuthSplitTest -Dmaven.main.skip=true test`
- [x] `mvn -pl paimon-core -Pfast-build -Dtest=SplitSerializerTest,QueryAuthSplitTest -Dmaven.main.skip=true test`
- [x] `mvn -pl paimon-core spotless:check`
- [x] `git diff --check`
- [x] JDK 8 `javac` compile for the changed main classes
## Notes
`FormatDataSplit` and object table splits are intentionally excluded from this protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add versioned split serializer [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8482:
URL: https://github.com/apache/paimon/pull/8482#discussion_r3530281537
##########
paimon-core/src/main/java/org/apache/paimon/table/source/QueryAuthSplit.java:
##########
@@ -54,10 +61,37 @@ public long rowCount() {
@Override
public OptionalLong mergedRowCount() {
- List<String> filter = authResult.filter();
- if (filter != null && !filter.isEmpty()) {
- return OptionalLong.empty();
+ if (authResult != null) {
+ List<String> filter = authResult.filter();
+ if (filter != null && !filter.isEmpty()) {
+ return OptionalLong.empty();
+ }
}
return split.mergedRowCount();
}
+
+ private void writeObject(ObjectOutputStream out) throws IOException {
+ serialize(new DataOutputViewStreamWrapper(out));
+ }
+
+ private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException {
Review Comment:
This breaks Java serialization compatibility for QueryAuthSplit. Old Flink checkpoints use the previous default object stream, but readObject now expects the new SplitSerializer format. I reproduced restore as EOFException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add versioned split serializer [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8482:
URL: https://github.com/apache/paimon/pull/8482#discussion_r3532880948
##########
paimon-core/src/main/java/org/apache/paimon/table/source/QueryAuthSplit.java:
##########
@@ -54,10 +61,37 @@ public long rowCount() {
@Override
public OptionalLong mergedRowCount() {
- List<String> filter = authResult.filter();
- if (filter != null && !filter.isEmpty()) {
- return OptionalLong.empty();
+ if (authResult != null) {
+ List<String> filter = authResult.filter();
+ if (filter != null && !filter.isEmpty()) {
+ return OptionalLong.empty();
+ }
}
return split.mergedRowCount();
}
+
+ private void writeObject(ObjectOutputStream out) throws IOException {
+ serialize(new DataOutputViewStreamWrapper(out));
+ }
+
+ private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException {
Review Comment:
This is expected. Firstly, cross version compatibility is not guaranteed, and secondly, QueryAuthSplit is generally not saved in Flink state.
However, I should indeed increase the serial Version UID to make the exception more apparent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add versioned split serializer [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8482:
URL: https://github.com/apache/paimon/pull/8482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add versioned split serializer [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8482:
URL: https://github.com/apache/paimon/pull/8482#issuecomment-4899628279
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org