You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/06/26 09:26:38 UTC

[I] Expose DataFrame-style write API (WriteBuilder / TableWrite / TableCommit) to Python [paimon-rust]

JunRuiLee opened a new issue, #414:
URL: https://github.com/apache/paimon-rust/issues/414

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.
   
   
   ### Motivation
   
   Companion to the read-path effort (#<READ_ISSUE>). PyPaimon's write path is
   still pure Python, while the Rust core already implements the full write
   pipeline in `crates/paimon`:
   `WriteBuilder → new_write() → TableWrite.write_arrow() / prepare_commit() →
   new_commit() → TableCommit.commit(messages)`.
   
   This pipeline is **already exercised in production by the SQL `INSERT INTO`
   path** (`crates/integrations/datafusion/src/physical_plan/sink.rs`), and
   `TableCommit` already handles snapshot-conflict retries
   (`commit_if_latest_snapshot` + retry/wait). It is simply **not exposed through
   `bindings/python`**.
   
   Goal: surface the existing Rust write API to Python so PyPaimon's batch write
   path can run on Rust, with the public Python API unchanged.
   
   Out of scope: streaming write, and merge-into/update/delete DML (already
   reachable via SQL).
   
   
   ## Scope (incremental PRs)
   
   - **PR 1** — Expose `WriteBuilder` + `TableWrite`: `new_batch_write_builder()`,
     `with_overwrite()`, `new_write()`, and `write_arrow(batch)` accepting a
     PyArrow RecordBatch (Arrow FFI, zero-copy in).
   - **PR 2** — Expose commit: `prepare_commit()` → serializable `PyCommitMessage`
     list, and `new_commit().commit(messages)` driving `TableCommit.commit`
     (reusing the existing conflict-retry logic).
   - **PR 3** (in `apache/paimon`, `[python]`) — Switch PyPaimon's batch write
     (`BatchTableWrite` / `BatchTableCommit`) to delegate to the Rust pipeline.
   
   PR 1–2 land here; PR 3 lands in the main repo and depends on a `pypaimon-rust`
   build on PyPI that includes the new bindings.
   
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose DataFrame-style write API (WriteBuilder / TableWrite / TableCommit) to Python [paimon-rust]

Posted by "JingsongLi (via GitHub) via issues" <is...@paimon.apache.org>.
JingsongLi closed issue #414: Expose DataFrame-style write API (WriteBuilder / TableWrite / TableCommit) to Python
URL: https://github.com/apache/paimon-rust/issues/414


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org