You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/09/27 13:03:25 UTC
[PR] [python] Support Parquet-only local caching [paimon]
XiaoHongbo-Hope opened a new pull request, #10225:
URL: https://github.com/apache/paimon/pull/10225
### Purpose
Add `blob-meta` to the Python local-cache whitelist for BLOB footers, row indexes, and ARRAY/MAP headers, keys and indexes. Cache exact ranges without pulling adjacent value bodies into the cache, sharing the existing memory/disk budget.
Include it in the Python default whitelist; local caching remains disabled by default. `data` continues to cache ordinary blocks. Rust native reads are outside this change.
Cache accounting includes estimated memory overhead or allocated disk bytes, with a 65,536-entry cap per manager. Disk reopen enforces the same limits. Oversized entries bypass the cache; evicted files are detached under the index lock and deleted outside it. BLOB body `readinto()` delegates directly to the underlying stream when supported.
### Design precedent
[RocksDB BlobDB](https://github.com/facebook/rocksdb/wiki/BlobDB#column-family-options) stores blob references in SST blocks and large values in separate blob files. Reference blocks use the block cache; value caching is configured through `blob_cache`. Its documentation prioritizes reference blocks over individual blob values.
This PR applies the same rationale to repeated BLOB reads: cache the information needed to locate values independently of their bodies. This also benefits training when payloads must still be downloaded. Paimon stores this information inside `.blob` files, so the reader caches exact metadata ranges and parses them to produce descriptors.
### Tests
250 tests and 60 subtests passed, including 1,000-row ARRAY/MAP capacity checks, entry limits, disk reopen, descriptor/body read ranges `BlobRef.readinto()`, slow-write cache hits and concurrent same-key publication, oversized MAP keys and slow eviction. Full Python Flake8 and diff checks passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support structured-data local cache selection [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10225:
URL: https://github.com/apache/paimon/pull/10225#discussion_r4323132331
##########
paimon-python/pypaimon/read/reader/format_pyarrow_reader.py:
##########
@@ -559,8 +576,7 @@ def __init__(self, file_io: FileIO, file_format: str, file_path: str,
and self._selected_shared_map_paths)):
import pyarrow.parquet as pq
# ParquetFile(filesystem=...) is unavailable in PyArrow 6.
- self._parquet_source = file_io.filesystem.open_input_file(
- file_path_for_pyarrow)
+ self._parquet_source = _open_pyarrow_input_file(file_io, file_path)
Review Comment:
[P2] Forward the known file size to the direct Parquet reader
`SplitRead` already supplies `DataFileMeta.file_size`, but this call drops it when row selection, bounded VARIANT or selected MAP reads open their separate Parquet source. The resulting `CachingInputStream` calls the delegate's `get_file_size()` before reading cached blocks. With the actual Jindo handler, the same known-size selected-row read makes 0 backend metadata requests without the cache wrapper and 1 with it.
I also reproduced this with a fully cached Parquet file: perform a full scan with `file_size` supplied, remove the source and clear the parsed-footer cache, then read again with the same `file_size` and `row_indices=[1]`. The full warm scan succeeds, but the selected-row scan raises `FileNotFoundError` while querying the source size. Forwarding the known size makes the selected-row scan succeed as well.
Please pass `file_size` to `_open_pyarrow_input_file` here and add coverage combining a known size with selected-row warm reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support structured-data local cache selection [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10225:
URL: https://github.com/apache/paimon/pull/10225#issuecomment-6414637696
Please rebase master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support BLOB metadata caching [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #10225: [python] Support BLOB metadata caching
URL: https://github.com/apache/paimon/pull/10225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support structured-data local cache selection [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10225:
URL: https://github.com/apache/paimon/pull/10225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support structured-data local cache selection [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10225:
URL: https://github.com/apache/paimon/pull/10225#issuecomment-6434364716
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org