You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/08/03 16:23:26 UTC

[I] DataFusion scan planning can exhaust memory by cloning split metadata [paimon-rust]

XiaoHongbo-Hope opened a new issue, #665:
URL: https://github.com/apache/paimon-rust/issues/665

   ## Bug
   
   For scans with many manifest entries and data files, the DataFusion planning path can retain multiple full copies of immutable split metadata:
   
   - Table scan planning builds a Vec<DataSplit> containing DataFileMeta values.
   - PaimonScanBuilder copies the plan with splits().to_vec(), which deep-clones the partition, bucket path, data files, deletion files, and row ranges.
   - Asynchronous readers clone splits again before execution.
   - Manifest reads collect every per-manifest result Vec before flattening them, and manifest ADD/DELETE merging allocates another output Vec.
   
   The resulting peak memory is a multiple of the planned metadata size and can exhaust a query worker during planning, before it returns any rows.
   
   ## Expected behavior
   
   Planning should transfer ownership where possible and share immutable split metadata across necessary clones. Manifest results should be merged incrementally and reuse their allocation without changing scan semantics.
   
   ## Compatibility requirements
   
   - Preserve the public DataSplit getters.
   - Preserve JSON and Java binary split serialization.
   - Preserve manifest merge and scan behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] DataFusion scan planning can exhaust memory by cloning split metadata [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #665: DataFusion scan planning can exhaust memory by cloning split metadata
URL: https://github.com/apache/paimon-rust/issues/665


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org