You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/07/01 04:10:30 UTC
[PR] [python][daft] Add read_blob for concurrent blob reads [paimon]
XiaoHongbo-Hope opened a new pull request, #8406:
URL: https://github.com/apache/paimon/pull/8406
### Purpose
### Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3511113131
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,36 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range. Thread-safe. ``length < 0`` = read to EOF (pread can't
+ express that, so seek + read)."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if length >= 0 and supports_pread(stream):
+ return pread(stream, length, offset)
+ stream.seek(offset)
+ return stream.read() if length < 0 else stream.read(length)
+ finally:
+ stream.close()
+
+
+def read_blobs_concurrent(file_io, blobs, parallelism):
+ """Read a list of Blobs concurrently via ThreadPool + pread."""
+ from concurrent.futures import ThreadPoolExecutor
+
+ def _read_one(blob):
+ if blob is None:
+ return None
+ return blob.to_data()
+
+ non_null = [b for b in blobs if b is not None]
+ if not non_null:
+ return [None] * len(blobs)
+ workers = min(parallelism, len(non_null))
+ with ThreadPoolExecutor(workers) as pool:
+ return list(pool.map(_read_one, blobs))
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3510718551
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,36 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range. Thread-safe. ``length < 0`` = read to EOF (pread can't
+ express that, so seek + read)."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if length >= 0 and supports_pread(stream):
+ return pread(stream, length, offset)
+ stream.seek(offset)
+ return stream.read() if length < 0 else stream.read(length)
+ finally:
+ stream.close()
+
+
+def read_blobs_concurrent(file_io, blobs, parallelism):
+ """Read a list of Blobs concurrently via ThreadPool + pread."""
+ from concurrent.futures import ThreadPoolExecutor
+
+ def _read_one(blob):
+ if blob is None:
+ return None
+ return blob.to_data()
+
+ non_null = [b for b in blobs if b is not None]
+ if not non_null:
+ return [None] * len(blobs)
+ workers = min(parallelism, len(non_null))
+ with ThreadPoolExecutor(workers) as pool:
+ return list(pool.map(_read_one, blobs))
Review Comment:
Do we need to do some IO merging? For example, if the current number of blob reads is greater than the concurrent number, we can look at the same file's blob and merge them together for reading if the intervals are not far apart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3511128169
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,36 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range. Thread-safe. ``length < 0`` = read to EOF (pread can't
+ express that, so seek + read)."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if length >= 0 and supports_pread(stream):
+ return pread(stream, length, offset)
+ stream.seek(offset)
+ return stream.read() if length < 0 else stream.read(length)
+ finally:
+ stream.close()
+
+
+def read_blobs_concurrent(file_io, blobs, parallelism):
+ """Read a list of Blobs concurrently via ThreadPool + pread."""
+ from concurrent.futures import ThreadPoolExecutor
+
+ def _read_one(blob):
+ if blob is None:
+ return None
+ return blob.to_data()
+
+ non_null = [b for b in blobs if b is not None]
+ if not non_null:
+ return [None] * len(blobs)
+ workers = min(parallelism, len(non_null))
+ with ThreadPoolExecutor(workers) as pool:
+ return list(pool.map(_read_one, blobs))
Review Comment:
> Do we need to do some IO merging? For example, if the current number of blob reads is greater than the concurrent number, we can look at the same file's blob and merge them together for reading if the intervals are not far apart.
Great idea, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3511113131
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,36 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range. Thread-safe. ``length < 0`` = read to EOF (pread can't
+ express that, so seek + read)."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if length >= 0 and supports_pread(stream):
+ return pread(stream, length, offset)
+ stream.seek(offset)
+ return stream.read() if length < 0 else stream.read(length)
+ finally:
+ stream.close()
+
+
+def read_blobs_concurrent(file_io, blobs, parallelism):
+ """Read a list of Blobs concurrently via ThreadPool + pread."""
+ from concurrent.futures import ThreadPoolExecutor
+
+ def _read_one(blob):
+ if blob is None:
+ return None
+ return blob.to_data()
+
+ non_null = [b for b in blobs if b is not None]
+ if not non_null:
+ return [None] * len(blobs)
+ workers = min(parallelism, len(non_null))
+ with ThreadPoolExecutor(workers) as pool:
+ return list(pool.map(_read_one, blobs))
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3510235022
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,35 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range from a file. Thread-safe."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if supports_pread(stream):
+ return pread(stream, length, offset)
Review Comment:
This breaks valid `BlobDescriptor.length == -1` descriptors. `OffsetInputStream` treats `-1` as “read to EOF”, but this path passes `-1` into `read_at`/`os.pread`, which raises (for example, local FileIO hits `OSError: [Errno 22] Invalid argument`). Please special-case negative lengths by seeking to `offset` and reading to EOF, or compute the remaining file size before using `pread`, and add coverage for `BlobDescriptor(..., length=-1)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3511124108
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,35 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
+ """Read a byte range from a file. Thread-safe."""
+ stream = file_io.new_input_stream(path)
+ try:
+ if supports_pread(stream):
+ return pread(stream, length, offset)
Review Comment:
> This breaks valid `BlobDescriptor.length == -1` descriptors. `OffsetInputStream` treats `-1` as “read to EOF”, but this path passes `-1` into `read_at`/`os.pread`, which raises (for example, local FileIO hits `OSError: [Errno 22] Invalid argument`). Please special-case negative lengths by seeking to `offset` and reading to EOF, or compute the remaining file size before using `pread`, and add coverage for `BlobDescriptor(..., length=-1)`.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support concurrent blob reads for raw and Daft APIs [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8406:
URL: https://github.com/apache/paimon/pull/8406#discussion_r3511398319
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -48,6 +48,123 @@ def pread(stream, length: int, offset: int) -> bytes:
return os.pread(stream.fileno(), length, offset)
+def read_file_range(file_io, path, offset, length):
Review Comment:
Can we add these methods into FileIO?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org