You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by "viiccwen (via GitHub)" <gi...@apache.org> on 2026/05/04 17:31:47 UTC
[PR] Feat(GPU): add single-node distributed amplitude scaffolding [mahout]
viiccwen opened a new pull request, #1296:
URL: https://github.com/apache/mahout/pull/1296
## Summary
This PR introduces the **first single-node multi-GPU scaffolding for distributed amplitude state construction in QDP.**
It does **not** attempt to solve the entire multi-GPU roadmap. Instead, it establishes the core substrate needed to plan, validate, and materialize amplitude state shards across multiple GPUs on a single host.
Closed #1295
## Execution Flow
```mermaid
sequenceDiagram
participant U as Upstream Caller
participant V as Validator
participant M as DeviceMesh
participant T as Topology
participant P as Planner
participant F as Feasibility Check
participant L as Layout
participant R as Materializer
U->>V: submit request
V-->>U: reject if invalid
V->>M: resolve devices
M->>T: load topology metadata
V->>P: planning input
T->>P: ordering / link hints
P-->>F: placement plan
F-->>U: reject if any shard infeasible
F->>L: validated placements
L->>R: materialize buffers
R-->>U: distributed state handle
```
This ordering is intentional:
- validate request first
- build the device mesh second
- plan shard placement third
- validate shard feasibility before allocation
- materialize buffers only after the layout is known to be valid
## What this PR adds
### Distributed GPU substrate
- adds `DeviceMesh` for validated multi-device construction
- adds `GpuTopology` and initial topology-aware ordering/gather heuristics
- adds `Communicator` with a host-coordinated fallback implementation
- adds distributed planning/runtime modules under `qdp-core/src/gpu/distributed`
### Planning and validation
- adds `PlacementRequest`, `PlacementPlan`, and `PlacementPlanner`
- adds `DistributedAmplitudePlan` for shard math and feasibility checks
- validates request/data before mesh creation for earlier, cleaner failures
- rejects malformed distributed layouts instead of silently truncating shard/device mappings
### Materialized distributed state
- adds `DistributedStateLayout`
- adds `DistributedStateVector` with per-shard buffers
- adds host copy helpers for shard inspection in tests and smoke coverage
### Runtime / diagnostics improvements
- improves device-aware allocation diagnostics
- propagates device ordinal into allocation / upload error paths
- hardens distributed helper logic against out-of-range topology-derived indices
### Tests
- adds dedicated tests for:
- communicator behavior
- topology behavior
- placement planner behavior
- distributed amplitude planning
- distributed runtime validation
- distributed engine behavior
- adds a two-GPU distributed amplitude smoke test in `gpu_api_workflow.rs`
## Non-goals
This PR does not include:
- NCCL integration
- peer-to-peer copy optimization
- multi-node support
- full gather/export APIs
- complete heterogeneous-capacity placement policies
- full performance tuning
## Why this shape
The main goal of this PR is to keep the architecture extensible.
Instead of baking distributed behavior directly into amplitude encoding, this PR separates:
- request validation
- mesh/topology discovery
- placement planning
- logical distributed layout
- materialized distributed state
- communication backend
That separation should make later work on uneven shards, topology-aware placement, NCCL, and additional encoders much easier.
## Validation
Test coverage added for:
- invalid input paths
- shard range validation
- planner policy behavior
- topology-based placement ordering
- malformed mesh/layout rejection
- distributed shard host-copy behavior
- distributed amplitude parity/smoke paths where GPUs are available
## Follow-ups
Likely next steps:
- improve real topology fidelity beyond placeholder link quality
- add gather/export APIs
- add higher-performance communication backends
- evaluate more advanced placement policies for heterogeneous devices
## Reference
- Pytorch Distributed Design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4780387661
unstale
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4775005436
This pull request has been automatically marked as stale because it has not had any activity for 30 days.
It will be closed in another 7 days if no further activity occurs. Thank you for your contribution. If you'd like to keep this open, leave any comment and the stale label will be removed.
You can always ask for help on the [Mahout dev mailing list](https://lists.apache.org/list.html?dev@mahout.apache.org) or in [GitHub Discussions](https://github.com/apache/mahout/discussions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-5358245779
@viiccwen any updates on this? I was hoping to including distributed QDP in 0.7.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4421943244
@viiccwen do you have a design doc for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-6306070764
This pull request has been automatically marked as stale because it has not had any activity for 30 days.
It will be closed in another 7 days if no further activity occurs. Thank you for your contribution. If you'd like to keep this open, leave any comment and the stale label will be removed.
You can always ask for help on the [Mahout dev mailing list](https://lists.apache.org/list.html?dev@mahout.apache.org) or in [GitHub Discussions](https://github.com/apache/mahout/discussions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4397471376
> @viiccwen I think since it's 3 PRs away and so big. Can we postpone it to the next release? It will be more mature at next release and we can think about its detail.
Sure, it'll be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4421962297
Also please solve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-5746794866
This pull request has been automatically marked as stale because it has not had any activity for 30 days.
It will be closed in another 7 days if no further activity occurs. Thank you for your contribution. If you'd like to keep this open, leave any comment and the stale label will be removed.
You can always ask for help on the [Mahout dev mailing list](https://lists.apache.org/list.html?dev@mahout.apache.org) or in [GitHub Discussions](https://github.com/apache/mahout/discussions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4814540008
This branch supports single-process, single-node multi-GPU shard materialization with zero-copy local shard views. It does not yet implement true MPI multi-rank execution or cross-rank GPU-resident zero-copy transport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #1296:
URL: https://github.com/apache/mahout/pull/1296#discussion_r3740132334
##########
qdp/qdp-core/src/gpu/encodings/amplitude.rs:
##########
@@ -762,7 +789,7 @@ impl AmplitudeEncoder {
input_ptr,
state_ptr_offset,
chunk_len,
- state_len,
+ chunk_state_len,
Review Comment:
This chunk-length fix (`state_len` -> `chunk_state_len`) is a legitimate, tested fix to pre-existing non-distributed code, but it's unrelated to this PR's stated distributed/MPI scaffolding scope. Consider splitting into its own PR for a cleaner history; fine to leave as-is if you'd rather not.
##########
qdp/qdp-core/src/gpu/distributed/shared.rs:
##########
@@ -0,0 +1,64 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::gpu::topology::GpuTopology;
+
+pub(crate) fn policy_device_ids(
Review Comment:
`policy_device_ids()` indexes an already topology-reordered `shard_device_ids` using an index computed against the mesh's original (pre-reorder) topology matrix. Using the mesh/topology from the existing test `sharded_capacity_prefers_topology_recommended_device_order` (device_ids `[10,11,12]`, topology-correct gather device is 11, per `plan.gather_device_id == Some(11)` in that test), this function returns `Some(10)` instead. Currently advisory-only (nothing consumes it yet), but it defeats the purpose of the topology-aware placement API, and no test exercises these two methods directly. Recommend deriving the recommendation the same way `PlacementPlan.gather_device_id` does (directly from the mesh), plus a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4396231538
@viiccwen I think since it's 3 PRs away. Can we postpone it to the next release? It will be more mature at next release and we can think about its detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat(GPU): add single-node distributed amplitude scaffolding [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4373882775
Nice one, will take a look on probably Thursday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4388917915
@ryankert01, after researching, I think the main gap is on the `lightning.gpu` integration boundary.
From what I can tell, the missing piece is not “can Mahout produce a GPU-resident state?” but “can `lightning.gpu` ingest an external GPU state buffer through a stable public interface without copying?”.
So my current estimate would be:
1. expose or formalize an external-state ingest path on the PennyLane / `lightning.gpu` side
2. add a Mahout-side bridge/adapter for `mahout.qdp -> lightning.gpu`
3. add end-to-end tests and examples for the workflow
So yes: for a proper zero-copy integration, I would expect roughly 3 PRs for a narrow MVP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][QDP] Add MPI-ready distributed amplitude execution scaffolding [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #1296:
URL: https://github.com/apache/mahout/pull/1296#issuecomment-4426793187
@viiccwen — coordinating on multi-GPU scope. I'm planning a `PR 19` for DDP rank-aware sharding in `QuantumDataLoader` (Feistel-based O(1) per-rank disjoint partitioning) as part of the QDP pipeline refactor.
Two questions before I start the work:
1. **Rough merge ETA for #1296?** Saw a CI run on 2026-05-11 so it's active — just want to know if it's days away or weeks away so I can decide whether PR 19 waits or ships single-node-first.
2. **`MPIContext` / topology-API surface you're shaping?** I'd like PR 19 to consume it directly rather than define a parallel abstraction. Even a rough sketch of the public types would help me design the loader's rank/world_size plumbing to match yours.
Happy to align PR 19 with #1296 either way — prefer to wait if it's close, otherwise build single-node-first and layer MPI on top after #1296 merges. Let me know whichever works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org