You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/08/27 10:35:14 UTC

[PR] [python][torch] Support distributed iterable dataset sharding [paimon]

XiaoHongbo-Hope opened a new pull request, #9429:
URL: https://github.com/apache/paimon/pull/9429

   ## What changed
   
   - Add opt-in distributed sharding for PyTorch streaming datasets.
   - Resolve rank and world size from explicit arguments, an initialized process group, or torchrun environment variables.
   - Assign splits with balanced rank-first and worker-second slicing.
   - Include rank and worker identity in shuffled dataset random state.
   - Keep distributed sharding disabled by default for applications that already shard splits.
   
   ## Validation
   
   - PyTorch tests: 40 passed.
   - CPU torchrun integration: 2 ranks x 2 DataLoader workers, complete split union with no overlap.
   - flake8, license header check, and git diff --check passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][torch] Support distributed iterable dataset sharding [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9429:
URL: https://github.com/apache/paimon/pull/9429#discussion_r3888762800


##########
docs/docs/pypaimon/pytorch.md:
##########
@@ -60,6 +60,30 @@ when it is false, it will read the full amount of data into memory.
 **`prefetch_concurrency`** (default: 1): In streaming row mode, controls
 reader threads per DataLoader worker. It has no effect in non-streaming mode.
 
+### Distributed Sharding
+
+Streaming reads shard splits across DDP ranks and DataLoader workers:
+
+```python
+dataset = table_read.to_torch(
+    splits,
+    streaming=True,
+    auto_detect_rank=True,
+)
+dataloader = DataLoader(dataset, batch_size=32, num_workers=2)

Review Comment:
   **[P1] Use a spawn-safe DataLoader in this DDP example**
   
   On Linux, leaving `multiprocessing_context` unset makes a multi-worker DataLoader use `fork`. PyTorch explicitly warns that DDP with NCCL (and Gloo over InfiniBand) is not fork-safe and is likely to deadlock. This is especially easy to miss because the new torchrun test uses `multiprocessing_context="spawn"`, so it does not exercise the documented path. Please set `multiprocessing_context="spawn"` (or `forkserver`) in the example and mention the usual guarded-entry-point requirement for spawn. See https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html.



##########
paimon-python/pypaimon/read/datasource/torch_dataset.py:
##########
@@ -40,6 +41,59 @@ def _share_epoch_with_torch_workers(value):
     return torch.tensor(value, dtype=torch.long).share_memory_()
 
 
+def _validate_distributed_context(rank: int, world_size: int):
+    if isinstance(rank, bool) or not isinstance(rank, int):
+        raise ValueError("rank must be an int")
+    if isinstance(world_size, bool) or not isinstance(world_size, int):
+        raise ValueError("world_size must be an int")
+    if world_size <= 0:
+        raise ValueError("world_size must be greater than 0")
+    if rank < 0 or rank >= world_size:
+        raise ValueError("rank must satisfy 0 <= rank < world_size")
+    return rank, world_size
+
+
+def _resolve_distributed_context(auto_detect_rank: bool):
+    if not isinstance(auto_detect_rank, bool):
+        raise ValueError("auto_detect_rank must be a bool")
+    if not auto_detect_rank:
+        return 0, 1
+
+    distributed = getattr(torch, "distributed", None)
+    if (
+        distributed is not None
+        and distributed.is_available()
+        and distributed.is_initialized()
+    ):
+        rank = distributed.get_rank()

Review Comment:
   **[P2] Shard against the DDP process group, not always the global world**
   
   Calling `get_rank()` and `get_world_size()` without a group always resolves the default global process group, while DDP may be constructed with `process_group=subgroup`. In subgroup or hybrid-parallel training, `auto_detect_rank=True` will therefore shard inputs across the global world instead of the replicas synchronized by DDP; a subgroup can see only part of its dataset, or model-parallel ranks can receive different inputs. Please either accept explicit sharding rank/world size resolved from the intended DDP group before serializing the dataset to workers, or document that auto-detection supports only whole-world DDP and require caller-side pre-sharding otherwise. See https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][torch] Support distributed iterable dataset sharding [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9429:
URL: https://github.com/apache/paimon/pull/9429


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][torch] Support distributed iterable dataset sharding [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9429:
URL: https://github.com/apache/paimon/pull/9429#issuecomment-5467491943

   I compared this design with the current PyTorch, Lance, Hugging Face Datasets, WebDataset, and Ray Data implementations. Overall, the direction in this PR looks sound, and I do not see an architectural blocker.
   
   The main reasons are:
   
   * A plain PyTorch `IterableDataset` must own worker-replica sharding itself (`get_worker_info()`); DDP does not shard inputs, and `DataLoader` rejects a custom sampler for an `IterableDataset`. Therefore, doing process-rank sharding in the dataset and then DataLoader-worker sharding is the correct layer. See the [PyTorch DataLoader source](https://github.com/pytorch/pytorch/blob/main/torch/utils/data/dataloader.py#L359-L369) and [DDP documentation](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html).
   * Hugging Face Datasets and WebDataset use the same broad hierarchy: first select the rank/node-local sources, then split those sources among DataLoader workers before opening/decoding them. See [HF distributed.py](https://github.com/huggingface/datasets/blob/main/src/datasets/distributed.py), [HF iterable_dataset.py](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py), and [WebDataset shardlists.py](https://github.com/webdataset/webdataset/blob/main/src/webdataset/shardlists.py).
   * Keeping `auto_detect_rank=False` by default is appropriate for Paimon. `TableRead.to_torch` receives an already-planned split list and cannot know whether the caller has pre-sharded it. Automatically sharding by default could therefore silently apply sharding twice. Systems such as Ray can safely auto-split because the trainer owns the complete dataset plan and all consumers; that is a different ownership boundary. See [Ray's dataset-shard API](https://docs.ray.io/en/latest/train/api/doc/ray.train.get_dataset_shard.html).
   * The explicit `sharding_rank` / `sharding_world_size` pair is important for DDP subgroups and hybrid parallelism. Resolving these values in the parent and serializing scalar context into spawned workers is also a good design.
   
   One useful comparison is Lance. Lance flattens distributed rank and DataLoader worker id into a global rank and supports fragment- and batch-level sharding. However, based on the current source, its distributed helper only observes a process group initialized in the current process. A spawned DataLoader worker does not retain the parent's initialized process group, while the deprecated explicit `rank/world_size` branch does not additionally shard among worker replicas. Therefore, as a source-level inference, Lance's iterable path does not transparently cover the full `spawn + DDP + num_workers > 0` combination. Paimon's captured parent rank followed by a separate worker split handles this combination more robustly. See [LanceDataset](https://github.com/lance-format/lance/blob/main/python/python/lance/torch/data.py), [Lance distributed helpers](https://github.com/lance-format/lance/blob/main/python/python/lance/torch/dist.py), and [Lance samplers](https://github.com/lance-format/la
 nce/blob/main/python/python/lance/sampler.py).
   
   I would keep the current exact-coverage policy as well: assign every split once, allow uneven rank lengths, and require DDP uneven-input handling. This is preferable to silently duplicating or dropping data. A few non-blocking follow-ups may still be worthwhile:
   
   1. Clarify that `set_epoch()` changes each rank's local buffer-shuffle order, but does not reshuffle split ownership across ranks as PyTorch `DistributedSampler` does.
   2. Mention that training loops with additional non-DDP collectives (for example, `SyncBatchNorm`) should use `model.join(throw_on_early_termination=True)`; plain `join()` can only shadow DDP's own collectives.
   3. Consider warning when the number of splits is smaller than `world_size * num_workers`, since some ranks/workers will be idle.
   4. If equal-step training becomes a requirement later, add an explicit policy such as preserve-all/drop-to-equal/repeat-to-equal rather than changing the current behavior implicitly. Similarly, fragment/split versus batch/weighted sharding can be added as an explicit strategy later without coupling it to rank detection.
   
   So my recommendation is to keep the current API boundary and rank -> worker -> reader hierarchy. The comparison with other implementations reinforces this design rather than suggesting that Paimon should copy Lance's defaults.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][torch] Support distributed iterable dataset sharding [paimon]

Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #9429:
URL: https://github.com/apache/paimon/pull/9429#issuecomment-5467683182

   > So my recommendation is to keep the current API boundary and rank -> worker -> reader hierarchy. The comparison with other implementations reinforces this design rather than suggesting that Paimon should copy Lance's defaults.
   
   Agreed, I also went back and forth on Lance’s sharding model before settling on this design.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org