You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hechao-ustc (via GitHub)" <gi...@apache.org> on 2026/07/18 06:37:56 UTC
[PR] docs(python): add Python Integration documentation page [paimon-rust]
hechao-ustc opened a new pull request, #538:
URL: https://github.com/apache/paimon-rust/pull/538
<!--
Thank you very much for contributing to Paimon Rust - we are happy that you want to help us improve it. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/paimon-rust/issues). Exceptions are made for typos in documentation or comments, which need no issue.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `cargo test` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Add a new Python Integration page to the Paimon Rust documentation, following the same format as the existing Go Integration page.
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
- Add docs/src/python-binding.md — Python Integration documentation
- Add Python Integration entry to docs/mkdocs.yml navigation
### Tests
<!-- List unit tests or integration cases to verify this change -->
N/A — documentation-only change.
### API and Format
<!-- Does this change affect API or storage format -->
N/A — no API or format changes.
### Documentation
<!-- Does this change introduce a new feature or require documentation updates -->
This PR is the documentation. The new page covers all Python binding APIs available at bindings/python/:
- Catalog creation (filesystem, OSS, REST)
- SQL Context (SQLContext) with DDL/DML support
- Reading tables (scan-then-read via ReadBuilder)
- Writing tables (write-then-commit via WriteBuilder)
- Column projection, limit, case sensitivity
- Filter push-down (lightweight dict predicate format, compound predicates, literal type mapping)
- Time travel (snapshot ID, timestamp, version, tag)
- Table inspection (snapshots, tags, partition stats)
- Python UDF registration
- Complete end-to-end example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(python): add Python Integration documentation page [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #538:
URL: https://github.com/apache/paimon-rust/pull/538#discussion_r3607904555
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,441 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
+
+## Installation
+
+```bash
+pip install pypaimon-rust
Review Comment:
`pypaimon-rust` does not declare `pyarrow` as a runtime dependency. In a clean environment, following this installation command causes subsequent queries to fail with `ModuleNotFoundError: No module named pyarrow`.
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,441 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
+
+## Installation
+
+```bash
+pip install pypaimon-rust
+```
+
+The pre-built native library is embedded in the package and automatically loaded at runtime — no manual build step is needed.
+
+## Creating a Catalog
+
+Use `PaimonCatalog` with a dictionary of options to create a catalog. The catalog type is determined by the `metastore` option (default: `filesystem`).
+
+```python
+from pypaimon_rust.datafusion import PaimonCatalog
+
+# Local filesystem
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+
+# List databases and tables
+print(catalog.list_databases())
+print(catalog.list_tables("default"))
+
+# Get a table handle
+table = catalog.get_table("default.my_table")
+```
+
+### Alibaba Cloud OSS
+
+```python
+catalog = PaimonCatalog({
+ "warehouse": "oss://bucket/warehouse",
+ "fs.oss.accessKeyId": "your-access-key-id",
+ "fs.oss.accessKeySecret": "your-access-key-secret",
+ "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com",
+})
+```
+
+### REST Catalog
+
+```python
+catalog = PaimonCatalog({
+ "metastore": "rest",
+ "uri": "http://localhost:8080",
+ "warehouse": "my_warehouse",
+})
+```
+
+## SQL Context
+
+`SQLContext` supports registering multiple Paimon catalogs and executing SQL queries with DataFusion.
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+# DDL and DML
+ctx.sql("CREATE SCHEMA paimon.my_db")
+ctx.sql("CREATE TABLE paimon.my_db.t (id INT, name STRING)")
+ctx.sql("INSERT INTO paimon.my_db.t VALUES (1, 'alice'), (2, 'bob')")
+
+# Query returns a list of PyArrow RecordBatches
+batches = ctx.sql("SELECT * FROM paimon.my_db.t")
+for batch in batches:
+ print(batch)
+```
+
+## Reading a Table
+
+Paimon Python uses a **scan-then-read** pattern: first scan the table to produce splits, then read data from those splits as PyArrow RecordBatches.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Create a read builder
+rb = table.new_read_builder()
+
+# Step 1: Scan — produces a Plan containing Splits
+scan = rb.new_scan()
+plan = scan.plan()
+splits = plan.splits()
+
+# Step 2: Read — consumes splits and returns PyArrow RecordBatches
+read = rb.new_read()
+batches = read.read(splits)
+
+for batch in batches:
+ print(batch)
+```
+
+Alternatively, read via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+batches = ctx.sql("SELECT id, name FROM paimon.default.my_table")
+for batch in batches:
+ print(batch)
+```
+
+## Writing to a Table
+
+Paimon Python uses a **write-then-commit** pattern: write PyArrow RecordBatches to a writer, prepare commit messages, then commit.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Build a batch matching the table schema
+batch = pa.record_batch(
+ [pa.array([1, 2, 3], pa.int32()), pa.array(["a", "b", "c"], pa.string())],
+ names=["id", "name"],
+)
+
+# Create a write builder (shared commit_user for writer and committer)
+wb = table.new_write_builder()
+
+# Write batches
+write = wb.new_write()
+write.write_arrow(batch)
+
+# Prepare commit messages
+messages = write.prepare_commit()
+
+# Commit
+wb.new_commit().commit(messages)
+```
+
+Alternatively, write via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+ctx.sql("INSERT INTO paimon.default.my_table VALUES (1, 'alice'), (2, 'bob')")
+```
+
+!!! warning "Schema Validation"
+ The input batch schema is strictly validated against the table schema: field count, order, names, and types must match exactly. A `ValueError` is raised on mismatch.
+
+!!! note "Write Builder Consistency"
+ The writer and committer must come from the same `WriteBuilder` — they share a `commit_user` for duplicate-commit detection. Passing messages from one builder's writer to another builder's committer will raise a `ValueError`.
+
+## Column Projection
+
+Use `with_projection` to select specific columns. Only the requested columns are read, reducing I/O.
+
+```python
+rb = table.new_read_builder()
+rb.with_projection(["id", "name"])
+
+# Continue with scan-then-read as above...
+```
+
+## Limit
+
+Use `with_limit` to cap the number of rows returned. A limit of `0` returns zero rows.
Review Comment:
`with_limit` is only a scan-planning hint and does not guarantee that at most the specified number of rows is returned. For example, `with_limit(1)` still returns all 3 rows when they belong to a single split.
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,441 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
+
+## Installation
+
+```bash
+pip install pypaimon-rust
+```
+
+The pre-built native library is embedded in the package and automatically loaded at runtime — no manual build step is needed.
+
+## Creating a Catalog
+
+Use `PaimonCatalog` with a dictionary of options to create a catalog. The catalog type is determined by the `metastore` option (default: `filesystem`).
+
+```python
+from pypaimon_rust.datafusion import PaimonCatalog
+
+# Local filesystem
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+
+# List databases and tables
+print(catalog.list_databases())
+print(catalog.list_tables("default"))
+
+# Get a table handle
+table = catalog.get_table("default.my_table")
+```
+
+### Alibaba Cloud OSS
+
+```python
+catalog = PaimonCatalog({
+ "warehouse": "oss://bucket/warehouse",
+ "fs.oss.accessKeyId": "your-access-key-id",
+ "fs.oss.accessKeySecret": "your-access-key-secret",
+ "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com",
+})
+```
+
+### REST Catalog
+
+```python
+catalog = PaimonCatalog({
+ "metastore": "rest",
+ "uri": "http://localhost:8080",
+ "warehouse": "my_warehouse",
+})
+```
+
+## SQL Context
+
+`SQLContext` supports registering multiple Paimon catalogs and executing SQL queries with DataFusion.
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+# DDL and DML
+ctx.sql("CREATE SCHEMA paimon.my_db")
+ctx.sql("CREATE TABLE paimon.my_db.t (id INT, name STRING)")
+ctx.sql("INSERT INTO paimon.my_db.t VALUES (1, 'alice'), (2, 'bob')")
+
+# Query returns a list of PyArrow RecordBatches
+batches = ctx.sql("SELECT * FROM paimon.my_db.t")
+for batch in batches:
+ print(batch)
+```
+
+## Reading a Table
+
+Paimon Python uses a **scan-then-read** pattern: first scan the table to produce splits, then read data from those splits as PyArrow RecordBatches.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Create a read builder
+rb = table.new_read_builder()
+
+# Step 1: Scan — produces a Plan containing Splits
+scan = rb.new_scan()
+plan = scan.plan()
+splits = plan.splits()
+
+# Step 2: Read — consumes splits and returns PyArrow RecordBatches
+read = rb.new_read()
+batches = read.read(splits)
+
+for batch in batches:
+ print(batch)
+```
+
+Alternatively, read via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+batches = ctx.sql("SELECT id, name FROM paimon.default.my_table")
+for batch in batches:
+ print(batch)
+```
+
+## Writing to a Table
+
+Paimon Python uses a **write-then-commit** pattern: write PyArrow RecordBatches to a writer, prepare commit messages, then commit.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Build a batch matching the table schema
+batch = pa.record_batch(
+ [pa.array([1, 2, 3], pa.int32()), pa.array(["a", "b", "c"], pa.string())],
+ names=["id", "name"],
+)
+
+# Create a write builder (shared commit_user for writer and committer)
+wb = table.new_write_builder()
+
+# Write batches
+write = wb.new_write()
+write.write_arrow(batch)
+
+# Prepare commit messages
+messages = write.prepare_commit()
+
+# Commit
+wb.new_commit().commit(messages)
+```
+
+Alternatively, write via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+ctx.sql("INSERT INTO paimon.default.my_table VALUES (1, 'alice'), (2, 'bob')")
+```
+
+!!! warning "Schema Validation"
+ The input batch schema is strictly validated against the table schema: field count, order, names, and types must match exactly. A `ValueError` is raised on mismatch.
+
+!!! note "Write Builder Consistency"
+ The writer and committer must come from the same `WriteBuilder` — they share a `commit_user` for duplicate-commit detection. Passing messages from one builder's writer to another builder's committer will raise a `ValueError`.
+
+## Column Projection
+
+Use `with_projection` to select specific columns. Only the requested columns are read, reducing I/O.
+
+```python
+rb = table.new_read_builder()
+rb.with_projection(["id", "name"])
+
+# Continue with scan-then-read as above...
+```
+
+## Limit
+
+Use `with_limit` to cap the number of rows returned. A limit of `0` returns zero rows.
+
+```python
+rb = table.new_read_builder()
+rb.with_limit(100)
+```
+
+## Case Sensitivity
+
+Use `with_case_sensitive` to control whether column-name matching in projections and predicates is case-sensitive. Defaults to `True` (exact match). Set to `False` for case-insensitive matching (ASCII case-folding).
+
+```python
+rb = table.new_read_builder()
+rb.with_case_sensitive(False)
+```
+
+## Filter Push-Down
+
+Filter push-down prunes data at two levels:
+
+1. **Scan planning** — skips partitions, buckets, and data files based on file-level statistics (min/max).
+2. **Read-side** — applies row-level filtering via Parquet native row filters for leaf predicates.
+
+!!! warning
+ Filter push-down is a **best-effort** optimization. The returned results may still contain rows that do not satisfy the filter condition. Callers should always apply residual filtering on the returned records to ensure correctness.
+
+### Predicate Format
+
+Predicates use a lightweight dictionary format. Each leaf node specifies a `method`, `field`, and `literals`:
+
+```python
+# Comparison predicates
+{"method": "equal", "field": "id", "literals": [1]} # id = 1
+{"method": "notEqual", "field": "name", "literals": ["bob"]} # name != "bob"
+{"method": "lessThan", "field": "id", "literals": [3]} # id < 3
+{"method": "lessOrEqual", "field": "id", "literals": [2]} # id <= 2
+{"method": "greaterThan", "field": "id", "literals": [1]} # id > 1
+{"method": "greaterOrEqual", "field": "id", "literals": [2]} # id >= 2
+
+# Null checks
+{"method": "isNull", "field": "name"} # name IS NULL
+{"method": "isNotNull", "field": "name"} # name IS NOT NULL
+
+# IN / NOT IN
+{"method": "in", "field": "id", "literals": [1, 2, 3]} # id IN (1, 2, 3)
+{"method": "notIn", "field": "name", "literals": ["x", "y"]} # name NOT IN ("x", "y")
+
+# String predicates
+{"method": "startsWith", "field": "name", "literals": ["al"]} # name LIKE 'al%'
+{"method": "endsWith", "field": "name", "literals": ["ce"]} # name LIKE '%ce'
+{"method": "contains", "field": "name", "literals": ["ic"]} # name LIKE '%ic%'
+{"method": "like", "field": "name", "literals": ["a%b%c"]} # name LIKE 'a%b%c'
+```
+
+### Applying Filters
+
+Pass a predicate dict to `with_filter` on the `ReadBuilder`:
+
+```python
+rb = table.new_read_builder()
+rb.with_filter({"method": "equal", "field": "id", "literals": [1]})
+
+# Continue with scan-then-read...
+```
+
+### Compound Predicates
+
+Combine predicates with `"and"` / `"or"` methods using a `"children"` list:
+
+```python
+# id >= 1 AND id <= 3
+rb.with_filter({
+ "method": "and",
+ "children": [
+ {"method": "greaterOrEqual", "field": "id", "literals": [1]},
+ {"method": "lessOrEqual", "field": "id", "literals": [3]},
+ ]
+})
+
+# (id = 1 OR id = 2) AND name = "alice"
+rb.with_filter({
+ "method": "and",
+ "children": [
+ {
+ "method": "or",
+ "children": [
+ {"method": "equal", "field": "id", "literals": [1]},
+ {"method": "equal", "field": "id", "literals": [2]},
+ ]
+ },
+ {"method": "equal", "field": "name", "literals": ["alice"]},
+ ]
+})
+```
+
+### Supported Literal Types
+
+Literal values are automatically converted from Python types based on the column's declared schema type:
+
+| Python Type | Paimon Type |
+|--------------------------------|----------------------|
+| `bool` | Bool |
+| `int` | TinyInt / SmallInt / Int / BigInt |
+| `int` / `float` | Float / Double |
+| `str` | String |
+| `datetime.date` | Date |
+| `datetime.time` (naive) | Time |
+| `datetime.datetime` (naive) | Timestamp |
+| `datetime.datetime` (aware) | LocalZonedTimestamp |
+| `decimal.Decimal` / `int` | Decimal |
+
+For temporal types, ensure the Python object matches the column type exactly:
+
+```python
+import datetime
+from decimal import Decimal
+
+# Date
+{"method": "equal", "field": "dt", "literals": [datetime.date(2024, 1, 1)]}
+
+# Decimal at scale 2 — int or Decimal accepted
+{"method": "equal", "field": "amount", "literals": [Decimal("123.45")]}
+
+# Naive datetime for TIMESTAMP (no timezone)
+{"method": "equal", "field": "ts", "literals": [datetime.datetime(2024, 1, 1, 12, 0, 0)]}
+
+# Aware datetime for TIMESTAMP WITH LOCAL TIME ZONE
+from zoneinfo import ZoneInfo
+{"method": "equal", "field": "ts_ltz", "literals": [
+ datetime.datetime(2024, 1, 1, 12, 0, 0, tzinfo=ZoneInfo("Asia/Shanghai"))
+]}
+```
+
+## Time Travel
+
+Use scan options on `new_read_builder` to read a table at a specific point in time:
+
+```python
+# By snapshot ID
+rb = table.new_read_builder({"scan.snapshot-id": "1"})
+
+# By timestamp (epoch millis)
+rb = table.new_read_builder({"scan.timestamp-millis": "1700000000000"})
+
+# By version
+rb = table.new_read_builder({"scan.version": "3"})
+
+# By tag name
+rb = table.new_read_builder({"scan.tag-name": "release-1.0"})
+```
+
+!!! warning
+ Only one time-travel selector may be set. Providing multiple selectors will raise a `ValueError`.
+
+## Table Inspection
+
+Inspect snapshots, tags, and partition statistics on a table:
+
+```python
+# Latest snapshot
+snap = table.latest_snapshot()
+if snap:
+ print(f"Snapshot {snap.id()} at {snap.commit_time_ms()}")
+ print(f" commit kind: {snap.commit_kind()}")
+ print(f" total records: {snap.total_record_count()}")
+
+# All snapshots (newest first)
+for snap in table.list_snapshots():
+ print(snap.id(), snap.commit_kind())
+
+# Tags
+for tag in table.list_tags():
+ print(tag.name(), tag.snapshot_id())
+
+# Partition stats
+for stat in table.partition_stats():
+ print(stat.partition(), stat.record_count(), stat.total_size_bytes())
+```
+
+## Python UDF
+
+Register Python scalar UDFs into a `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+import pyarrow as pa
+
+ctx = SQLContext()
+
+def add_one(args):
+ # args is a tuple of PyArrow Arrays
+ arr = args[0]
+ return pa.array([v.as_py() + 1 for v in arr], type=pa.int32())
+
+ctx.register_udf(
+ ctx.udf(add_one, [pa.int32()], pa.int32(), "immutable", name="add_one")
Review Comment:
This UDF example does not run: `SQLContext` has no `udf` method, and a single-argument UDF receives a PyArrow Array rather than a tuple. Consequently, `args[0]` produces a Scalar and triggers a runtime error.
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,441 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
+
+## Installation
+
+```bash
+pip install pypaimon-rust
+```
+
+The pre-built native library is embedded in the package and automatically loaded at runtime — no manual build step is needed.
+
+## Creating a Catalog
+
+Use `PaimonCatalog` with a dictionary of options to create a catalog. The catalog type is determined by the `metastore` option (default: `filesystem`).
+
+```python
+from pypaimon_rust.datafusion import PaimonCatalog
+
+# Local filesystem
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+
+# List databases and tables
+print(catalog.list_databases())
+print(catalog.list_tables("default"))
+
+# Get a table handle
+table = catalog.get_table("default.my_table")
+```
+
+### Alibaba Cloud OSS
+
+```python
+catalog = PaimonCatalog({
+ "warehouse": "oss://bucket/warehouse",
+ "fs.oss.accessKeyId": "your-access-key-id",
+ "fs.oss.accessKeySecret": "your-access-key-secret",
+ "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com",
+})
+```
+
+### REST Catalog
+
+```python
+catalog = PaimonCatalog({
+ "metastore": "rest",
+ "uri": "http://localhost:8080",
+ "warehouse": "my_warehouse",
+})
+```
+
+## SQL Context
+
+`SQLContext` supports registering multiple Paimon catalogs and executing SQL queries with DataFusion.
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+# DDL and DML
+ctx.sql("CREATE SCHEMA paimon.my_db")
+ctx.sql("CREATE TABLE paimon.my_db.t (id INT, name STRING)")
+ctx.sql("INSERT INTO paimon.my_db.t VALUES (1, 'alice'), (2, 'bob')")
+
+# Query returns a list of PyArrow RecordBatches
+batches = ctx.sql("SELECT * FROM paimon.my_db.t")
+for batch in batches:
+ print(batch)
+```
+
+## Reading a Table
+
+Paimon Python uses a **scan-then-read** pattern: first scan the table to produce splits, then read data from those splits as PyArrow RecordBatches.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Create a read builder
+rb = table.new_read_builder()
+
+# Step 1: Scan — produces a Plan containing Splits
+scan = rb.new_scan()
+plan = scan.plan()
+splits = plan.splits()
+
+# Step 2: Read — consumes splits and returns PyArrow RecordBatches
+read = rb.new_read()
+batches = read.read(splits)
+
+for batch in batches:
+ print(batch)
+```
+
+Alternatively, read via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+batches = ctx.sql("SELECT id, name FROM paimon.default.my_table")
+for batch in batches:
+ print(batch)
+```
+
+## Writing to a Table
+
+Paimon Python uses a **write-then-commit** pattern: write PyArrow RecordBatches to a writer, prepare commit messages, then commit.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Build a batch matching the table schema
+batch = pa.record_batch(
+ [pa.array([1, 2, 3], pa.int32()), pa.array(["a", "b", "c"], pa.string())],
+ names=["id", "name"],
+)
+
+# Create a write builder (shared commit_user for writer and committer)
+wb = table.new_write_builder()
+
+# Write batches
+write = wb.new_write()
+write.write_arrow(batch)
+
+# Prepare commit messages
+messages = write.prepare_commit()
+
+# Commit
+wb.new_commit().commit(messages)
+```
+
+Alternatively, write via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+ctx.sql("INSERT INTO paimon.default.my_table VALUES (1, 'alice'), (2, 'bob')")
+```
+
+!!! warning "Schema Validation"
+ The input batch schema is strictly validated against the table schema: field count, order, names, and types must match exactly. A `ValueError` is raised on mismatch.
+
+!!! note "Write Builder Consistency"
+ The writer and committer must come from the same `WriteBuilder` — they share a `commit_user` for duplicate-commit detection. Passing messages from one builder's writer to another builder's committer will raise a `ValueError`.
+
+## Column Projection
+
+Use `with_projection` to select specific columns. Only the requested columns are read, reducing I/O.
+
+```python
+rb = table.new_read_builder()
+rb.with_projection(["id", "name"])
+
+# Continue with scan-then-read as above...
+```
+
+## Limit
+
+Use `with_limit` to cap the number of rows returned. A limit of `0` returns zero rows.
+
+```python
+rb = table.new_read_builder()
+rb.with_limit(100)
+```
+
+## Case Sensitivity
+
+Use `with_case_sensitive` to control whether column-name matching in projections and predicates is case-sensitive. Defaults to `True` (exact match). Set to `False` for case-insensitive matching (ASCII case-folding).
Review Comment:
The filter predicate is constructed using the current case-sensitivity setting when `with_filter` is called. Calling `with_case_sensitive(False)` afterward does not affect an existing predicate, but this ordering constraint is not documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(python): add Python Integration documentation page [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #538:
URL: https://github.com/apache/paimon-rust/pull/538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(python): add Python Integration documentation page [paimon-rust]
Posted by "hechao-ustc (via GitHub)" <gi...@apache.org>.
hechao-ustc commented on PR #538:
URL: https://github.com/apache/paimon-rust/pull/538#issuecomment-5010525257
@QuakeWang All changes have been made as requested, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(python): add Python Integration documentation page [paimon-rust]
Posted by "hechao-ustc (via GitHub)" <gi...@apache.org>.
hechao-ustc commented on PR #538:
URL: https://github.com/apache/paimon-rust/pull/538#issuecomment-5013927279
@QuakeWang Addressed comments, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(python): add Python Integration documentation page [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #538:
URL: https://github.com/apache/paimon-rust/pull/538#discussion_r3608020714
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,455 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
Review Comment:
The supported-platform list does not match the published artifacts: the release workflow builds a Windows wheel, and a win_amd64 wheel is available on PyPI.
##########
docs/src/python-binding.md:
##########
@@ -0,0 +1,455 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Python Integration
+
+The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## Prerequisites
+
+- Python 3.10 or later
+- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64)
+
+## Installation
+
+```bash
+pip install pypaimon-rust pyarrow
+```
+
+The pre-built native library is embedded in the package and automatically loaded at runtime — no manual build step is needed. [PyArrow](https://arrow.apache.org/docs/python/) is a required peer dependency and must be installed separately.
+
+## Creating a Catalog
+
+Use `PaimonCatalog` with a dictionary of options to create a catalog. The catalog type is determined by the `metastore` option (default: `filesystem`).
+
+```python
+from pypaimon_rust.datafusion import PaimonCatalog
+
+# Local filesystem
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+
+# List databases and tables
+print(catalog.list_databases())
+print(catalog.list_tables("default"))
+
+# Get a table handle
+table = catalog.get_table("default.my_table")
+```
+
+### Alibaba Cloud OSS
+
+```python
+catalog = PaimonCatalog({
+ "warehouse": "oss://bucket/warehouse",
+ "fs.oss.accessKeyId": "your-access-key-id",
+ "fs.oss.accessKeySecret": "your-access-key-secret",
+ "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com",
+})
+```
+
+### REST Catalog
+
+```python
+catalog = PaimonCatalog({
+ "metastore": "rest",
+ "uri": "http://localhost:8080",
+ "warehouse": "my_warehouse",
+})
+```
+
+## SQL Context
+
+`SQLContext` supports registering multiple Paimon catalogs and executing SQL queries with DataFusion.
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+# DDL and DML
+ctx.sql("CREATE SCHEMA paimon.my_db")
+ctx.sql("CREATE TABLE paimon.my_db.t (id INT, name STRING)")
+ctx.sql("INSERT INTO paimon.my_db.t VALUES (1, 'alice'), (2, 'bob')")
+
+# Query returns a list of PyArrow RecordBatches
+batches = ctx.sql("SELECT * FROM paimon.my_db.t")
+for batch in batches:
+ print(batch)
+```
+
+## Reading a Table
+
+Paimon Python uses a **scan-then-read** pattern: first scan the table to produce splits, then read data from those splits as PyArrow RecordBatches.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Create a read builder
+rb = table.new_read_builder()
+
+# Step 1: Scan — produces a Plan containing Splits
+scan = rb.new_scan()
+plan = scan.plan()
+splits = plan.splits()
+
+# Step 2: Read — consumes splits and returns PyArrow RecordBatches
+read = rb.new_read()
+batches = read.read(splits)
+
+for batch in batches:
+ print(batch)
+```
+
+Alternatively, read via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+batches = ctx.sql("SELECT id, name FROM paimon.default.my_table")
+for batch in batches:
+ print(batch)
+```
+
+## Writing to a Table
+
+Paimon Python uses a **write-then-commit** pattern: write PyArrow RecordBatches to a writer, prepare commit messages, then commit.
+
+```python
+import pyarrow as pa
+from pypaimon_rust.datafusion import PaimonCatalog
+
+catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"})
+table = catalog.get_table("default.my_table")
+
+# Build a batch matching the table schema
+batch = pa.record_batch(
+ [pa.array([1, 2, 3], pa.int32()), pa.array(["a", "b", "c"], pa.string())],
+ names=["id", "name"],
+)
+
+# Create a write builder (shared commit_user for writer and committer)
+wb = table.new_write_builder()
+
+# Write batches
+write = wb.new_write()
+write.write_arrow(batch)
+
+# Prepare commit messages
+messages = write.prepare_commit()
+
+# Commit
+wb.new_commit().commit(messages)
+```
+
+Alternatively, write via SQL using `SQLContext`:
+
+```python
+from pypaimon_rust.datafusion import SQLContext
+
+ctx = SQLContext()
+ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"})
+
+ctx.sql("INSERT INTO paimon.default.my_table VALUES (1, 'alice'), (2, 'bob')")
+```
+
+!!! warning "Schema Validation"
+ The input batch schema is strictly validated against the table schema: field count, order, names, and types must match exactly. A `ValueError` is raised on mismatch.
+
+!!! note "Write Builder Consistency"
+ The writer and committer must come from the same `WriteBuilder` — they share a `commit_user` for duplicate-commit detection. Passing messages from one builder's writer to another builder's committer will raise a `ValueError`.
+
+## Column Projection
+
+Use `with_projection` to select specific columns. Only the requested columns are read, reducing I/O.
+
+```python
+rb = table.new_read_builder()
+rb.with_projection(["id", "name"])
+
+# Continue with scan-then-read as above...
+```
+
+## Limit
+
+Use `with_limit` to set a hint for the number of rows returned. A limit of `0` returns zero rows.
+
+```python
+rb = table.new_read_builder()
+rb.with_limit(100)
+```
+
+!!! warning
+ `with_limit` is a scan-planning hint, not an exact row cap. When all rows fall within a single split, the entire split is returned regardless of the limit value. Callers should apply application-level limiting if an exact upper bound is required.
+
+## Case Sensitivity
+
+Use `with_case_sensitive` to control whether column-name matching in projections and predicates is case-sensitive. Defaults to `True` (exact match). Set to `False` for case-insensitive matching (ASCII case-folding).
+
+```python
+rb = table.new_read_builder()
+rb.with_case_sensitive(False)
+```
+
+!!! note
+ `with_case_sensitive` must be called **before** `with_filter` to affect predicate construction. The predicate is built using the case-sensitivity setting at the time `with_filter` is invoked; changing it afterward has no effect on an already-constructed predicate.
+
+## Filter Push-Down
+
+Filter push-down prunes data at two levels:
+
+1. **Scan planning** — skips partitions, buckets, and data files based on file-level statistics (min/max).
+2. **Read-side** — applies row-level filtering via Parquet native row filters for leaf predicates.
+
+!!! warning
+ Filter push-down is a **best-effort** optimization. The returned results may still contain rows that do not satisfy the filter condition. Callers should always apply residual filtering on the returned records to ensure correctness.
Review Comment:
The filter semantics described here do not match the current implementation. Scan pruning can be conservative, but the read path applies exact residual filtering and does not return rows that fail the predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org