You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/09/12 08:24:51 UTC
[PR] [python] Read LeRobot video datasets for training [paimon]
XiaoHongbo-Hope opened a new pull request, #9757:
URL: https://github.com/apache/paimon/pull/9757
### Purpose
Support map-style training from video-backed LeRobot v3 table groups imported by `load_from_lerobot()`. This is the read-side follow-up to #9494 and #9498.
### Changes
- decode `.video` frame descriptors inside `PaimonLeRobotDataset`
- reuse the existing per-worker bounded video decoder cache
- prefer TorchCodec and fall back to PyAV; `video_backend` can select either
- apply existing episode selection, delta windows, padding, transforms, and `return_uint8` behavior to video features
- keep encoded video payloads lazy during planning and indexed row lookup
TorchCodec is the preferred random-access path. The PyAV fallback scans forward and rewinds for backward frame access.
### Tests
- 78 LeRobot and video tests passed
- real MP4 import and descriptor-backed map-style reads
- delta windows, Episode padding, decoder reuse, and spawn DataLoader workers
- flake8, Python 3.6/3.11 compile, and `git diff --check` passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Read LeRobot video datasets for training [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9757:
URL: https://github.com/apache/paimon/pull/9757#issuecomment-5652218133
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Read LeRobot video datasets for training [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9757:
URL: https://github.com/apache/paimon/pull/9757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Read LeRobot video datasets for training [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #9757:
URL: https://github.com/apache/paimon/pull/9757#discussion_r3998942839
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1164,6 +1224,173 @@ def _image_tensor(payload, feature, return_uint8=False):
return tensor.div_(255) if normalize else tensor
+def _video_tensor(frame, feature, return_uint8=False):
+ import torch
+
+ if not torch.is_tensor(frame):
+ raise ValueError("LeRobot video decoder must return a Torch tensor.")
+ expected_shape = _feature_shape(feature, "video")
+ if len(expected_shape) != 3:
+ raise ValueError("LeRobot video feature must have three dimensions.")
+ names = feature.get("names") or []
+ output_shape = expected_shape if names and names[0] in (
+ "channel", "channels"
+ ) else expected_shape[2:] + expected_shape[:2]
+ if tuple(frame.shape) != output_shape:
+ raise ValueError(
+ "LeRobot video frame has shape %s, expected %s."
+ % (tuple(frame.shape), output_shape)
+ )
+ if frame.dtype == torch.uint8 and not return_uint8:
+ return frame.float().div_(255)
+ return frame
+
+
+def _open_video_decoder(stream, backend=None):
+ if backend in (None, "torchcodec"):
+ try:
+ return _open_torchcodec_decoder(stream)
+ except (ImportError, OSError, RuntimeError):
+ if backend == "torchcodec":
+ raise
+ stream.seek(0)
+ return _PyAVVideoDecoder(stream)
+
+
+def _open_torchcodec_decoder(stream):
+ try:
+ from torchcodec.decoders import VideoDecoder
+ except (ImportError, RuntimeError) as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires TorchCodec from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ try:
+ return VideoDecoder(stream, seek_mode="exact")
+ except TypeError:
+ # TorchCodec 0.2 accepts bytes but not seekable file-like objects.
+ stream.seek(0)
+ return VideoDecoder(stream.read(), seek_mode="exact")
+
+
+class _PyAVVideoDecoder:
+
+ # Reuse common overlapping delta windows without retaining a whole video.
+ _FRAME_CACHE_SIZE = 8
+
+ def __init__(self, stream):
+ try:
+ import av
+ except ImportError as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires PyAV from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ self._container = av.open(stream)
+ self._stream = self._container.streams.video[0]
+ self._next_index = 0
+ self._timestamps = []
+ self._keyframes = []
+ self._cache = OrderedDict()
+ self._frames = iter(self._container.decode(self._stream))
+
+ def __getitem__(self, index):
+ index = operator.index(index)
+ if index < 0:
+ raise IndexError("Video frame index %d is out of range." % index)
+ frame = self._cache.pop(index, None)
+ if frame is not None:
+ self._cache[index] = frame
+ return self._tensor(frame)
+
+ at_frontier = self._next_index == len(self._timestamps)
+ if index != self._next_index and not (
+ at_frontier and index >= self._next_index):
+ self._seek(index)
Review Comment:
Fixed in 4cfabc7b34. A cold non-zero read now demuxes packet timestamps/keyframe flags once, builds the complete ordinal/keyframe index, seeks to the target GOP, and decodes only from that keyframe. The new regression requests frame 95 from a fresh 120-frame decoder and verifies only frames 90–95 are decoded; I also reran random reads on a real MP4 with B-frames.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Read LeRobot video datasets for training [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #9757:
URL: https://github.com/apache/paimon/pull/9757#discussion_r3998942839
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1164,6 +1224,173 @@ def _image_tensor(payload, feature, return_uint8=False):
return tensor.div_(255) if normalize else tensor
+def _video_tensor(frame, feature, return_uint8=False):
+ import torch
+
+ if not torch.is_tensor(frame):
+ raise ValueError("LeRobot video decoder must return a Torch tensor.")
+ expected_shape = _feature_shape(feature, "video")
+ if len(expected_shape) != 3:
+ raise ValueError("LeRobot video feature must have three dimensions.")
+ names = feature.get("names") or []
+ output_shape = expected_shape if names and names[0] in (
+ "channel", "channels"
+ ) else expected_shape[2:] + expected_shape[:2]
+ if tuple(frame.shape) != output_shape:
+ raise ValueError(
+ "LeRobot video frame has shape %s, expected %s."
+ % (tuple(frame.shape), output_shape)
+ )
+ if frame.dtype == torch.uint8 and not return_uint8:
+ return frame.float().div_(255)
+ return frame
+
+
+def _open_video_decoder(stream, backend=None):
+ if backend in (None, "torchcodec"):
+ try:
+ return _open_torchcodec_decoder(stream)
+ except (ImportError, OSError, RuntimeError):
+ if backend == "torchcodec":
+ raise
+ stream.seek(0)
+ return _PyAVVideoDecoder(stream)
+
+
+def _open_torchcodec_decoder(stream):
+ try:
+ from torchcodec.decoders import VideoDecoder
+ except (ImportError, RuntimeError) as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires TorchCodec from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ try:
+ return VideoDecoder(stream, seek_mode="exact")
+ except TypeError:
+ # TorchCodec 0.2 accepts bytes but not seekable file-like objects.
+ stream.seek(0)
+ return VideoDecoder(stream.read(), seek_mode="exact")
+
+
+class _PyAVVideoDecoder:
+
+ # Reuse common overlapping delta windows without retaining a whole video.
+ _FRAME_CACHE_SIZE = 8
+
+ def __init__(self, stream):
+ try:
+ import av
+ except ImportError as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires PyAV from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ self._container = av.open(stream)
+ self._stream = self._container.streams.video[0]
+ self._next_index = 0
+ self._timestamps = []
+ self._keyframes = []
+ self._cache = OrderedDict()
+ self._frames = iter(self._container.decode(self._stream))
+
+ def __getitem__(self, index):
+ index = operator.index(index)
+ if index < 0:
+ raise IndexError("Video frame index %d is out of range." % index)
+ frame = self._cache.pop(index, None)
+ if frame is not None:
+ self._cache[index] = frame
+ return self._tensor(frame)
+
+ at_frontier = self._next_index == len(self._timestamps)
+ if index != self._next_index and not (
+ at_frontier and index >= self._next_index):
+ self._seek(index)
Review Comment:
Fixed in 4cfabc7b34. A cold non-zero read now demuxes packet timestamps/keyframe flags once, builds the complete ordinal/keyframe index, seeks to the target GOP, and decodes only from that keyframe. The new regression requests frame 95 from a fresh 120-frame decoder and verifies only frames 90–95 are decoded; I also reran random reads on a real MP4 with B-frames.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Read LeRobot video datasets for training [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9757:
URL: https://github.com/apache/paimon/pull/9757#discussion_r3998881264
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1164,6 +1224,173 @@ def _image_tensor(payload, feature, return_uint8=False):
return tensor.div_(255) if normalize else tensor
+def _video_tensor(frame, feature, return_uint8=False):
+ import torch
+
+ if not torch.is_tensor(frame):
+ raise ValueError("LeRobot video decoder must return a Torch tensor.")
+ expected_shape = _feature_shape(feature, "video")
+ if len(expected_shape) != 3:
+ raise ValueError("LeRobot video feature must have three dimensions.")
+ names = feature.get("names") or []
+ output_shape = expected_shape if names and names[0] in (
+ "channel", "channels"
+ ) else expected_shape[2:] + expected_shape[:2]
+ if tuple(frame.shape) != output_shape:
+ raise ValueError(
+ "LeRobot video frame has shape %s, expected %s."
+ % (tuple(frame.shape), output_shape)
+ )
+ if frame.dtype == torch.uint8 and not return_uint8:
+ return frame.float().div_(255)
+ return frame
+
+
+def _open_video_decoder(stream, backend=None):
+ if backend in (None, "torchcodec"):
+ try:
+ return _open_torchcodec_decoder(stream)
+ except (ImportError, OSError, RuntimeError):
+ if backend == "torchcodec":
+ raise
+ stream.seek(0)
+ return _PyAVVideoDecoder(stream)
+
+
+def _open_torchcodec_decoder(stream):
+ try:
+ from torchcodec.decoders import VideoDecoder
+ except (ImportError, RuntimeError) as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires TorchCodec from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ try:
+ return VideoDecoder(stream, seek_mode="exact")
+ except TypeError:
+ # TorchCodec 0.2 accepts bytes but not seekable file-like objects.
+ stream.seek(0)
+ return VideoDecoder(stream.read(), seek_mode="exact")
+
+
+class _PyAVVideoDecoder:
+
+ # Reuse common overlapping delta windows without retaining a whole video.
+ _FRAME_CACHE_SIZE = 8
+
+ def __init__(self, stream):
+ try:
+ import av
+ except ImportError as error:
+ raise ImportError(
+ "Video-backed PaimonLeRobotDataset requires PyAV from "
+ "'pypaimon[lerobot]'."
+ ) from error
+ self._container = av.open(stream)
+ self._stream = self._container.streams.video[0]
+ self._next_index = 0
+ self._timestamps = []
+ self._keyframes = []
+ self._cache = OrderedDict()
+ self._frames = iter(self._container.decode(self._stream))
+
+ def __getitem__(self, index):
+ index = operator.index(index)
+ if index < 0:
+ raise IndexError("Video frame index %d is out of range." % index)
+ frame = self._cache.pop(index, None)
+ if frame is not None:
+ self._cache[index] = frame
+ return self._tensor(frame)
+
+ at_frontier = self._next_index == len(self._timestamps)
+ if index != self._next_index and not (
+ at_frontier and index >= self._next_index):
+ self._seek(index)
Review Comment:
[P2] Avoid decoding the full video prefix on cold random reads
When PyAV is selected explicitly or used as the fallback, a fresh decoder skips seeking and decodes every frame from 0 through the first requested index. The collator retains only eight videos per camera, so shuffled training over more files can repeatedly discard the learned timestamp/keyframe index and pay this cost again.
On a 6,000-frame, 62 MB H.264 MP4, reading frame 5,999 decoded all 6,000 frames and took about 1.4 seconds. Scanning packet timestamps took 12–15 ms, and seeking to the final GOP and decoding its 30 frames took about 8 ms.
Please build or reuse an ordinal/keyframe index from demuxed packets so cold random reads can seek to the required GOP while preserving exact frame ordinals. The existing importer already uses packet timestamps to determine frame ordinals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org