You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/27 23:05:46 UTC

[I] Spark row-based parquet reads regenerate the same row projection for every file [hudi]

yihua opened a new issue, #20109:
URL: https://github.com/apache/hudi/issues/20109

   Spark reads of Hudi tables generate an `UnsafeProjection` for every file on the row-based parquet path, and the file group reader format generates its output projection per file or file slice. Spark caches the compiled class, but it regenerates the source and splits the expressions for every projection before the lookup. On wide nested schemas this is several milliseconds per file and a large share of executor CPU, although the projection is the same for most files of a scan.
   
   Proposal: reuse the generated projection across files that need the same one, keyed by everything the generation reads, without sharing a projection between iterators or threads.
   
   part of #20064
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org