You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/04 08:48:43 UTC
[PR] Align MAP shared-shredding writes and lifecycle with Java [paimon-rust]
JingsongLi opened a new pull request, #1027:
URL: https://github.com/apache/paimon-rust/pull/1027
### Purpose
Closes #1026.
Complete the Rust core support needed to enable PyPaimon Native MAP shared-shredding writes and updates. The baseline ignores placement policies, infers width from the first row, bypasses shredding for PK files, and emits duplicate Arrow schemas that PyArrow resolves differently from arrow-rs.
### Brief change log
- Implement Java's plain, sequential and LRU allocators, with LRU as the default. Values for duplicate keys come from the last input occurrence; overflow duplicates follow Java's map semantics.
- Select physical column counts using Java's completed-file window (20 files, p90, slack), starting from max-columns. Publish widths only after a successful close. Append writers await pending MAP file closes before choosing the next width; ordinary rolling remains asynchronous.
- Share format context across rolled files. PK data and input changelog use independent contexts scoped to each buffer flush, matching Java's writer-factory lifetime.
- Route PK output through the shared format writer and keep statistics scoped to logical value fields.
- Emit exactly one final ARROW:schema footer entry containing shredding metadata, readable consistently by PyArrow and arrow-rs.
- Validate shared-shredding schema/options at schema creation/evolution and writer opening. Keep Rust's existing 16384-column reader bound consistent with writer configuration by rejecting oversized configurations before writing.
### Tests
- `cargo test --locked -p paimon --lib`: 3608 passed, 6 ignored.
- `cargo clippy --locked -p paimon --all-targets --features fulltext,vortex -- -D warnings`: passed.
- `cargo fmt --all -- --check`: passed.
- Regression coverage includes Java allocator examples, LRU eviction and metadata, p90/slack/window boundaries, invalid options, append/PK rolling, independent data/changelog histories, per-flush reset, value statistics, and a single final Arrow schema.
- A companion PyPaimon PR exercises native and Python cross-reads, selected MAP keys, nested values, duplicate keys, data-evolution updates and upserts against a locally rebuilt native package.
### API and Format
No public Rust API changes or new file layout version. Uses Java's existing PIP-43 physical MAP layout and metadata. Configured shared-shredding PK files now use the requested physical layout. Default placement follows current Java (LRU).
### Documentation
The companion PyPaimon PR documents the newly enabled Native MAP write/update path. Implementation comments identify the Java reference classes and context lifetime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align MAP writes and Variant shredding factories with Java [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #1027:
URL: https://github.com/apache/paimon-rust/pull/1027#discussion_r4177014320
##########
crates/paimon/src/arrow/format/mod.rs:
##########
@@ -470,111 +470,179 @@ fn supported_write_formats() -> Vec<&'static str> {
]
}
-/// Create a format writer that streams directly to storage.
+/// Reusable factory for one rolling writer, mirroring Java FormatWriterFactory.
+/// Output and compression belong to each file; schema/options belong to the factory.
+#[async_trait]
+pub(crate) trait FormatWriterFactory: Send + Sync {
+ async fn create_writer(
+ &self,
+ output: &OutputFile,
+ compression: &str,
+ ) -> crate::Result<Box<dyn FormatFileWriter>>;
+
+ /// Some plan factories need the previous close callback before the next file.
+ fn needs_completed_file_stats(&self) -> bool {
+ false
+ }
+}
+
+/// Build a factory once per rolling writer. Statistics describe logical value
+/// fields independently of a shredding plan's physical schema (notably for PK files).
+pub(crate) fn create_format_writer_factory(
+ file_format: &str,
+ schema: SchemaRef,
+ zstd_level: i32,
+ file_io: Option<FileIO>,
+ write_fields: Option<&[DataField]>,
+ format_options: Option<&HashMap<String, String>>,
+ stats_fields: Option<&[DataField]>,
+) -> crate::Result<Arc<dyn FormatWriterFactory>> {
+ let file_format = file_format.rsplit('.').next().unwrap_or_default();
+ if file_format.eq_ignore_ascii_case("parquet") {
Review Comment:
This factory only wraps Parquet, while `spec::map_shredding::validate` still explicitly accepts `orc` and Java's `OrcFileFormat` advertises `MAP_SHARED_SHREDDING` as supported. With `file.format=orc` and MAP shared-shredding enabled, Rust falls through to `PlainFormatWriterFactory`, so it writes the logical MAP layout without the shredding conversion/footer metadata (and the ORC read path has no corresponding MAP assembly). This silently violates the configured layout and is inconsistent with Java. Please either implement the ORC shredding writer/read path or reject ORC for MAP shared-shredding in validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align MAP writes and Variant shredding factories with Java [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1027:
URL: https://github.com/apache/paimon-rust/pull/1027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align MAP writes and Variant shredding factories with Java [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #1027:
URL: https://github.com/apache/paimon-rust/pull/1027#discussion_r4177379662
##########
crates/paimon/src/arrow/format/mod.rs:
##########
@@ -470,111 +470,179 @@ fn supported_write_formats() -> Vec<&'static str> {
]
}
-/// Create a format writer that streams directly to storage.
+/// Reusable factory for one rolling writer, mirroring Java FormatWriterFactory.
+/// Output and compression belong to each file; schema/options belong to the factory.
+#[async_trait]
+pub(crate) trait FormatWriterFactory: Send + Sync {
+ async fn create_writer(
+ &self,
+ output: &OutputFile,
+ compression: &str,
+ ) -> crate::Result<Box<dyn FormatFileWriter>>;
+
+ /// Some plan factories need the previous close callback before the next file.
+ fn needs_completed_file_stats(&self) -> bool {
+ false
+ }
+}
+
+/// Build a factory once per rolling writer. Statistics describe logical value
+/// fields independently of a shredding plan's physical schema (notably for PK files).
+pub(crate) fn create_format_writer_factory(
+ file_format: &str,
+ schema: SchemaRef,
+ zstd_level: i32,
+ file_io: Option<FileIO>,
+ write_fields: Option<&[DataField]>,
+ format_options: Option<&HashMap<String, String>>,
+ stats_fields: Option<&[DataField]>,
+) -> crate::Result<Arc<dyn FormatWriterFactory>> {
+ let file_format = file_format.rsplit('.').next().unwrap_or_default();
+ if file_format.eq_ignore_ascii_case("parquet") {
Review Comment:
Thanks, this is addressed: unsupported ORC MAP shared-shredding is now rejected before writing, with coverage for schema validation, direct writers, and append/PK writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org