(mahout) branch main updated: docs(qdp): sync python-api page with current qumat_qdp (#1287)

[hc...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ryankert01 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/mahout.git


The following commit(s) were added to refs/heads/main by this push:
     new 1fab161dc docs(qdp): sync python-api page with current qumat_qdp (#1287)
1fab161dc is described below

commit 1fab161dcc05df992a548749213e53e750380107
Author: Ryan Huang <hc...@apache.org>
AuthorDate: Sun Apr 26 02:57:04 2026 +0800

    docs(qdp): sync python-api page with current qumat_qdp (#1287)
    
    * docs(qdp): sync python-api page with current qumat_qdp
    
    * docs(qdp): address Copilot review on python-api page
    
    - Clarify that force_backend only mutates get_backend()/BACKEND and is
      not consulted by QdpBenchmark, QuantumDataLoader, or
      RustQdpEngine/NativeQuantumTensor availability; document the
      Backend | None signature.
    - Add CUDA tensor-input caveat: zero-copy CUDA tensors do not currently
      support phase encoding (CUDA_ENCODING_METHODS in src/constants.rs).
    - Add missing float32_pipeline=False parameter to the
      run_throughput_pipeline_py signature, matching the pyo3 binding in
      qdp-python/src/lib.rs.
    - Disambiguate the loader iteration table: the "rust" path yields the
      native _qdp.QuantumTensor (re-exported as NativeQuantumTensor), not
      the QuantumTensor facade alias for QdpTensor.
    
    
    ---------
---
 docs/qdp/python-api.md | 313 ++++++++++++++++++++++++++++++++++---------------
 1 file changed, 216 insertions(+), 97 deletions(-)

diff --git a/docs/qdp/python-api.md b/docs/qdp/python-api.md
index e0efd72c8..eb106faa6 100644
--- a/docs/qdp/python-api.md
+++ b/docs/qdp/python-api.md
@@ -5,111 +5,191 @@ sidebar_label: Python API
 
 # QDP Python API (qumat_qdp)
 
-Public Python API for QDP: GPU-accelerated encoding, benchmark helpers, and a batch iterator for training or evaluation loops.
+`qumat_qdp` is the user-facing Python package for QDP. It exposes:
 
-## Overview
-
-The **qumat_qdp** package wraps the native extension `_qdp` and adds:
+- A unified encoding facade via `QdpEngine`
+- Benchmark and loader builders
+- Backend-selection helpers
+- Optional access to native `_qdp` types when the Rust extension is installed
 
-- **Encoding:** `QdpEngine` and `QuantumTensor` for encoding classical data into quantum states and zero-copy DLPack integration.
-- **Benchmark:** `QdpBenchmark` for throughput/latency runs (full pipeline in Rust, GIL released).
-- **Data loader:** `QuantumDataLoader` for iterating encoded batches one at a time (`for qt in loader:`).
+## Overview
 
-Import from the package:
+Import from the package root:
 
 ```python
 from qumat_qdp import (
+    BACKEND,
+    Backend,
+    LatencyResult,
+    NativeQuantumTensor,
+    QdpBenchmark,
     QdpEngine,
+    QdpTensor,
+    QuantumDataLoader,
     QuantumTensor,
-    QdpBenchmark,
+    RustQdpEngine,
     ThroughputResult,
-    LatencyResult,
-    QuantumDataLoader,
+    TritonAmdEngine,
+    force_backend,
+    is_triton_amd_available,
     run_throughput_pipeline_py,
 )
 ```
 
-**Requirements:** Linux with NVIDIA GPU (CUDA). Loader and pipeline helpers are stubs on other platforms and raise `RuntimeError`.
+Key exports:
 
----
+- `QdpEngine`: unified encode entry point for `backend="cuda"` and `backend="amd"`.
+- `QdpBenchmark`: benchmark builder for the Rust pipeline or explicit PyTorch reference backend.
+- `QuantumDataLoader`: batch iterator builder for synthetic or file-backed inputs.
+- `QdpTensor` / `QuantumTensor`: thin DLPack facade type.
+- `Backend`, `BACKEND`, `force_backend`: backend detection and override helpers for the `_qdp` / PyTorch-reference selection layer.
+- `TritonAmdEngine`, `is_triton_amd_available`: direct AMD-route entry points.
+- `RustQdpEngine`, `NativeQuantumTensor`: native `_qdp` exports when the extension is available.
+
+`BACKEND` is the backend-selection snapshot captured when `qumat_qdp` is imported:
+
+- `Backend.RUST_CUDA`: `_qdp` is available
+- `Backend.NONE`: no auto-detected backend is available
+- `Backend.PYTORCH`: only when selected explicitly before import-time evaluation
+
+PyTorch is not used as an automatic fallback. The two backend-selection surfaces are independent:
+
+- `force_backend(backend: Backend | None)` only affects what `get_backend()` returns (and, transitively, the cached `BACKEND` snapshot at import time). Pass `None` to restore auto-detection. It does **not** switch which implementation `QdpBenchmark` or `QuantumDataLoader` use, and it does **not** change the availability of `RustQdpEngine` / `NativeQuantumTensor` (those depend on whether `_qdp` could be imported).
+- To run the PyTorch reference pipeline through the builders, call `.backend("pytorch")` on `QdpBenchmark` or `QuantumDataLoader` explicitly.
+
+If you need the current override state after import time, call `get_backend()` from `qumat_qdp._backend` rather than relying on the exported `BACKEND` constant.
+
+## Backend Model
+
+`qumat_qdp` has two related backend-selection surfaces:
+
+| Surface | Selector | Values |
+|--------|----------|--------|
+| Unified engine routing | `QdpEngine(..., backend=...)` | `"cuda"`, `"amd"` |
+| Builder fallback/reference routing | `.backend(...)` on `QdpBenchmark` / `QuantumDataLoader` | `"rust"`, `"pytorch"` |
+
+These selectors are intentionally different:
+
+- `QdpEngine(..., backend="cuda")` routes to the native `_qdp` CUDA engine.
+- `QdpEngine(..., backend="amd")` routes to the Triton AMD implementation.
+- `QdpBenchmark(...).backend("rust")` uses the Rust pipeline.
+- `QdpBenchmark(...).backend("pytorch")` uses the pure-PyTorch reference implementation.
+- `QuantumDataLoader(...).backend("rust")` uses the Rust-backed iterator.
+- `QuantumDataLoader(...).backend("pytorch")` uses the explicit PyTorch reference iterator.
 
 ## Encoding API
 
 ### QdpEngine
 
-GPU encoder. Constructor and main methods:
+Unified encoding facade.
 
-**`QdpEngine(device_id=0, precision="float32")`**
+**Constructor**
 
-- `device_id` (int): CUDA device ID.
-- `precision` (str): `"float32"` or `"float64"`.
-- Raises `RuntimeError` on init failure or unsupported precision.
+`QdpEngine(device_id=0, precision="float32", backend="cuda")`
 
-**`encode(data, num_qubits, encoding_method="amplitude") -> QuantumTensor`**
+- `device_id` (int): target device index
+- `precision` (str): `"float32"` or `"float64"`
+- `backend` (str): `"cuda"` or `"amd"`
 
-- `data`: list of floats, 1D/2D NumPy array (float64, C-contiguous), PyTorch tensor (CPU/CUDA), or file path (`.parquet`, `.arrow`, `.feather`, `.npy`, `.pt`, `.pth`, `.pb`).
-- `num_qubits` (int): Number of qubits.
-- `encoding_method` (str): `"amplitude"` | `"angle"` | `"basis"` | `"iqp"` | `"iqp-z"`.
-- Returns a DLPack-compatible tensor; use `torch.from_dlpack(qtensor)`. Shape `[batch_size, 2^num_qubits]`.
+**Method**
 
-**`create_synthetic_loader(total_batches, batch_size=64, num_qubits=16, encoding_method="amplitude", seed=None)`**
+`encode(data, num_qubits, encoding_method="amplitude")`
 
-- Returns an iterator that yields one `QuantumTensor` per batch. GIL is released during each encode. Linux/CUDA only.
+- `data`: Python list, NumPy array, PyTorch tensor, or other backend-supported input
+- `num_qubits` (int): number of qubits
+- `encoding_method` (str): depends on selected route
 
-### QuantumTensor
+Route support summary:
 
-DLPack wrapper for a GPU quantum state.
+| Route | Supported methods |
+|--------|-------------------|
+| `QdpEngine(..., backend="cuda")` | `amplitude`, `angle`, `basis`, `phase`, `iqp`, `iqp-z` |
+| `QdpEngine(..., backend="amd")` | `amplitude`, `angle`, `basis` |
 
-- **`__dlpack__(stream=None)`:** Returns a DLPack PyCapsule (single use).
-- **`__dlpack_device__()`:** Returns `(device_type, device_id)`; CUDA is `(2, gpu_id)`.
+**CUDA tensor-input caveat:** when `data` is provided as a zero-copy CUDA `torch.Tensor`, supported encoding methods are currently limited to `amplitude`, `angle`, `basis`, `iqp`, and `iqp-z`. In that input path, `phase` is not currently supported — pass a CPU tensor, NumPy array, or Python list to use `phase` on the CUDA route.
 
-If not consumed, memory is freed when the object is dropped; if consumed (e.g. by PyTorch), ownership transfers to the consumer.
+Result type notes:
 
----
+- The CUDA route returns the native `_qdp` tensor object when `_qdp` is available.
+- The AMD route currently returns a `torch.Tensor` from the Triton implementation.
+- `QdpTensor` / `QuantumTensor` provide a DLPack facade for backend-native tensor producers.
 
-## Benchmark API
+Direct AMD usage is also available:
+
+```python
+from qumat_qdp import QdpEngine, TritonAmdEngine
+
+router = QdpEngine(backend="amd", device_id=0, precision="float32")
+engine = TritonAmdEngine(device_id=0, precision="float32")
+```
+
+Use `is_triton_amd_available()` to check whether the Triton AMD runtime is available before selecting that route.
+
+### QdpTensor and QuantumTensor
+
+`QdpTensor` is a thin DLPack facade; `QuantumTensor` is an alias.
+
+Available methods:
+
+- `__dlpack__(stream=None)`
+- `__dlpack_device__()`
+- `to_torch()`
+
+Example:
+
+```python
+import torch
+from qumat_qdp import QdpEngine
+
+qt = QdpEngine(device_id=0, backend="cuda").encode(
+    [[1.0, 0.0, 0.0, 0.0]],
+    num_qubits=2,
+    encoding_method="amplitude",
+)
+tensor = torch.from_dlpack(qt)
+```
 
-Runs the full encode pipeline in Rust (warmup + timed loop) with GIL released. No Python-side loop.
+## Benchmark API
 
-### QdpBenchmark
+`QdpBenchmark` is a builder for throughput and latency runs.
 
-Builder; chain methods then call `run_throughput()` or `run_latency()`.
+**Constructor**
 
-**Constructor:** `QdpBenchmark(device_id=0)`
+`QdpBenchmark(device_id=0)`
 
-**Chainable methods:**
+**Chainable methods**
 
 | Method | Description |
 |--------|-------------|
-| `qubits(n)` | Number of qubits. |
-| `encoding(method)` | `"amplitude"` \| `"angle"` \| `"basis"`. |
-| `batches(total, size=64)` | Total batches and batch size. |
-| `prefetch(n)` | No-op (API compatibility). |
-| `warmup(n)` | Warmup batch count. |
-
-**`run_throughput() -> ThroughputResult`**
+| `qubits(n)` | Set number of qubits. |
+| `encoding(method)` | Set encoding method. |
+| `batches(total, size=64)` | Set total batches and batch size. |
+| `prefetch(n)` | No-op for API compatibility. |
+| `warmup(n)` | Set warmup batches. |
+| `backend(name)` | Select `"rust"` or `"pytorch"`. |
 
-- Requires `qubits` and `batches` to be set.
-- Returns `ThroughputResult` with `duration_sec`, `vectors_per_sec`.
-- Raises `ValueError` if config missing; `RuntimeError` if pipeline unavailable.
+Backend notes:
 
-**`run_latency() -> LatencyResult`**
+- `"rust"` is the default backend.
+- `"pytorch"` is explicit and intended as a reference path.
+- The Rust benchmark path is the native pipeline used by `run_throughput_pipeline_py`.
+- The PyTorch benchmark path is implemented in Python and is useful for fallback testing and comparisons.
 
-- Same pipeline; returns `LatencyResult` with `duration_sec`, `latency_ms_per_vector`.
+Encoding support summary:
 
-### Result types
+| Benchmark backend | Supported methods |
+|--------|-------------------|
+| `"rust"` | `amplitude`, `angle`, `basis` |
+| `"pytorch"` | `amplitude`, `angle`, `basis`, `iqp` |
 
-| Type | Fields |
-|------|--------|
-| `ThroughputResult` | `duration_sec`, `vectors_per_sec` |
-| `LatencyResult` | `duration_sec`, `latency_ms_per_vector` |
+Treat `phase` and `iqp-z` as direct-encode methods today rather than benchmark-helper features.
 
-### Example
+Example:
 
 ```python
-from qumat_qdp import QdpBenchmark, ThroughputResult, LatencyResult
+from qumat_qdp import QdpBenchmark
 
-result = (
+rust_result = (
     QdpBenchmark(device_id=0)
     .qubits(16)
     .encoding("amplitude")
@@ -117,44 +197,70 @@ result = (
     .warmup(2)
     .run_throughput()
 )
-print(result.vectors_per_sec)
 
-lat = (
+ref_result = (
     QdpBenchmark(device_id=0)
+    .backend("pytorch")
     .qubits(16)
-    .encoding("amplitude")
+    .encoding("iqp")
     .batches(100, size=64)
-    .run_latency()
+    .run_throughput()
 )
-print(lat.latency_ms_per_vector)
 ```
 
----
+Result types:
 
-## Data Loader API
+| Type | Fields |
+|--------|--------|
+| `ThroughputResult` | `duration_sec`, `vectors_per_sec` |
+| `LatencyResult` | `duration_sec`, `latency_ms_per_vector` |
 
-Iterate over encoded batches one at a time. Each batch is a `QuantumTensor`; encoding runs in Rust with GIL released per batch.
+## Data Loader API
 
-### QuantumDataLoader
+`QuantumDataLoader` is a builder for encoded batch iteration.
 
-Builder for a synthetic-data loader. Calling `iter(loader)` (or `for qt in loader`) creates the Rust-backed iterator.
+**Constructor**
 
-**Constructor:**
 `QuantumDataLoader(device_id=0, num_qubits=16, batch_size=64, total_batches=100, encoding_method="amplitude", seed=None)`
 
-**Chainable methods:**
+**Chainable methods**
 
 | Method | Description |
 |--------|-------------|
-| `qubits(n)` | Number of qubits. |
-| `encoding(method)` | `"amplitude"` \| `"angle"` \| `"basis"`. |
-| `batches(total, size=64)` | Total batches and batch size. |
-| `source_synthetic(total_batches=None)` | Synthetic data (default); optional override for total batches. |
-| `seed(s)` | RNG seed for reproducibility. |
+| `qubits(n)` | Set number of qubits. |
+| `encoding(method)` | Set encoding method. |
+| `batches(total, size=64)` | Set total batches and batch size. |
+| `source_synthetic(total_batches=None)` | Use synthetic data. |
+| `source_file(path, streaming=False)` | Use a file-backed source. |
+| `seed(s)` | Set reproducible synthetic-data seed. |
+| `null_handling(policy)` | Set `"fill_zero"` or `"reject"`. |
+| `backend(name)` | Select `"rust"` or `"pytorch"`. |
+
+File source notes:
+
+- `source_file(path, streaming=False)` accepts local paths and also accepts remote `s3://...` / `gs://...` paths when QDP is built with the optional `remote-io` feature.
+- Remote URL query strings and fragments are rejected.
+- `streaming=True` currently supports `.parquet` only.
+- The Rust backend supports broader file formats and streaming loaders.
+- The explicit PyTorch backend supports synthetic data plus `.npy`, `.pt`, and `.pth` files only.
+
+Iteration behavior depends on backend:
+
+| Loader backend | Iteration yields |
+|--------|------------------|
+| `"rust"` | `NativeQuantumTensor` / `_qdp.QuantumTensor` (the native Rust extension type, not the `QuantumTensor` facade alias) |
+| `"pytorch"` | `torch.Tensor` |
+
+Encoding support summary:
+
+| Loader backend | Supported methods |
+|--------|-------------------|
+| `"rust"` | documented for `amplitude`, `angle`, `basis` |
+| `"pytorch"` | `amplitude`, `angle`, `basis`, `iqp` |
 
-**Iteration:** `for qt in loader:` yields `QuantumTensor` of shape `[batch_size, 2^num_qubits]`. Consume once per tensor, e.g. `torch.from_dlpack(qt)`.
+`phase` and `iqp-z` are not documented loader-first methods.
 
-### Example
+Example:
 
 ```python
 from qumat_qdp import QuantumDataLoader
@@ -162,44 +268,57 @@ import torch
 
 loader = (
     QuantumDataLoader(device_id=0)
-    .qubits(16)
-    .encoding("amplitude")
-    .batches(100, size=64)
+    .backend("pytorch")
+    .qubits(4)
+    .encoding("iqp")
+    .batches(5, size=8)
     .source_synthetic()
 )
 
-for qt in loader:
-    batch = torch.from_dlpack(qt)  # [batch_size, 2^16]
-    # use batch ...
+for batch in loader:
+    assert isinstance(batch, torch.Tensor)
 ```
 
----
+Rust-backed file example:
 
-## Low-level: run_throughput_pipeline_py
+```python
+from qumat_qdp import QuantumDataLoader
+import torch
 
-Runs the full pipeline in Rust with GIL released. Used by `QdpBenchmark`; can be called directly.
+loader = (
+    QuantumDataLoader(device_id=0)
+    .qubits(8)
+    .encoding("amplitude")
+    .batches(10, size=32)
+    .source_file("dataset.parquet", streaming=True)
+    .null_handling("fill_zero")
+)
 
-**Signature:**
-`run_throughput_pipeline_py(device_id=0, num_qubits=16, batch_size=64, total_batches=100, encoding_method="amplitude", warmup_batches=0, seed=None) -> tuple[float, float, float]`
+for qt in loader:
+    batch = torch.from_dlpack(qt)
+```
 
-**Returns:** `(duration_sec, vectors_per_sec, latency_ms_per_vector)`.
+## Low-level Rust Pipeline Helper
 
-**Raises:** `RuntimeError` on failure or when not available (e.g. non-Linux).
+`run_throughput_pipeline_py(...)` is the low-level native helper used by the Rust benchmark path.
 
----
+Signature:
 
-## Backward compatibility
+`run_throughput_pipeline_py(device_id=0, num_qubits=16, batch_size=64, total_batches=100, encoding_method="amplitude", warmup_batches=0, seed=None, float32_pipeline=False)`
 
-`benchmark/api.py` and `benchmark/loader.py` re-export from `qumat_qdp`. Prefer:
+Returns a tuple:
 
-- `from qumat_qdp import QdpBenchmark, ThroughputResult, LatencyResult`
-- `from qumat_qdp import QuantumDataLoader`
+`(duration_sec, vectors_per_sec, latency_ms_per_vector)`
 
-Benchmark scripts add the project root to `sys.path`, so from the `qdp-python` directory you can run:
+This helper is only available when `_qdp` is installed.
+
+## Backward Compatibility
+
+`benchmark/api.py` and `benchmark/loader.py` continue to re-export the modern `qumat_qdp` API. Prefer importing from `qumat_qdp` directly.
+
+From `qdp/qdp-python`, benchmark scripts can still be run with:
 
 ```bash
 uv run python benchmark/run_pipeline_baseline.py
 uv run python benchmark/benchmark_loader_throughput.py
 ```
-
-without setting `PYTHONPATH`.