You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "dwangatt (via GitHub)" <gi...@apache.org> on 2026/09/25 01:45:15 UTC
[PR] [core] Route overwrite writes by partition bucket layout [paimon]
dwangatt opened a new pull request, #10170:
URL: https://github.com/apache/paimon/pull/10170
## Summary
This is a focused follow-up extracted from #9370, following the merged partition-layout scan prerequisite (#10052).
This PR makes partition bucket layout a core write contract:
- adds `PartitionBucketMapping` to resolve the current bucket count for each partition;
- adds write APIs that carry the routing bucket count:
`write(partition, bucket, totalBuckets, data)`;
- propagates the mapping into write restore so empty buckets recover the partition's active layout;
- ensures core overwrite writes can route and stamp files using the target partition layout;
- rejects the legacy `write(partition, bucket, data)` path when
`bucket.per-partition-count-enabled=true` on a partitioned table.
The legacy path only carries a bucket id and cannot establish which bucket count was used for hashing. Rejecting it prevents a caller from silently routing a row with the table-level bucket count while the target partition uses a different layout.
## Scope
This PR intentionally does not expose complete engine support for the option:
- Flink routing and end-to-end integration will be added in a follow-up PR.
- Spark will reject writes to tables with per-partition bucket counts until it has a safe routing implementation.
- Documentation and end-to-end tests will follow with the Flink integration.
## Verification
- `mvn -q -pl paimon-core -DskipTests compile`
- `mvn -q -pl paimon-core spotless:check`
- `git diff --check`
The focused restore test was updated to cover legacy-path rejection, stale-layout rejection, and a successful explicit partition-layout write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Route overwrite writes by partition bucket layout [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10170:
URL: https://github.com/apache/paimon/pull/10170#issuecomment-5831687516
Closing this PR under the end-to-end production-value criterion for this review pass. The PR introduces a public `bucket.per-partition-count-enabled` option and changes the core write contract, but its own scope statement leaves Flink routing, Spark's safe-write handling, configuration documentation, and engine-level end-to-end tests to later PRs. As submitted, there is no supported engine write path that demonstrates a partition being rescaled and then written/read safely with the new layout. The legacy core write path is deliberately rejected when the option is enabled, so merging this prerequisite alone would expose an option that users cannot safely use end to end.
I checked the full 23-file diff and ran the focused core suite after packaging the codegen loader: `PartitionBucketMappingTest`, `FixedBucketWriteSelectorTest`, `FixedBucketRowKeyExtractorTest`, `FileSystemWriteRestoreTest`, and `FileStoreCommitTest` passed (97 tests). This verifies the core pieces but does not close the engine-level gap. The new option also is absent from generated config docs, as noted in the existing review. The latest JDK 8/11 Core CI jobs stop in `S3FileIOTest` because the MinIO image cannot be fetched, which is unrelated to this feature.
Please bring the core routing contract back together with at least one complete engine write/read path, explicit behavior for the other engines, generated option docs, and a rescale-then-write/read integration test. That would make the production behavior reviewable as a whole.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Route overwrite writes by partition bucket layout [paimon]
Posted by "Akash3121 (via GitHub)" <gi...@apache.org>.
Akash3121 commented on code in PR #10170:
URL: https://github.com/apache/paimon/pull/10170#discussion_r4100758270
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -156,6 +156,16 @@ public class CoreOptions implements Serializable {
.withDescription(
"Whether to ignore the order of the buckets when reading data from an append-only table.");
+ public static final ConfigOption<Boolean> BUCKET_PER_PARTITION_COUNT_ENABLED =
Review Comment:
`ConfigOptionsDocsCompletenessITCase` discovers this new `CoreOptions` entry, but `bucket.per-partition-count-enabled` is absent from the generated configuration docs. The focused test currently fails with: `Option bucket.per-partition-count-enabled in class org.apache.paimon.CoreOptions is not documented`. This also explains the failing Core and integrations jobs on both JDK 8 and JDK 11. Please regenerate the configuration documentation according to `paimon-docs/README.md` (or explicitly exclude the option if it is intentionally internal) so the required completeness check passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Route overwrite writes by partition bucket layout [paimon]
Posted by "dwangatt (via GitHub)" <gi...@apache.org>.
dwangatt commented on PR #10170:
URL: https://github.com/apache/paimon/pull/10170#issuecomment-5880866831
Raised this PR(https://github.com/apache/paimon/pull/10297) to include the full functionality of adaptive bucketing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Route overwrite writes by partition bucket layout [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #10170: [core] Route overwrite writes by partition bucket layout
URL: https://github.com/apache/paimon/pull/10170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org