[I] Record index entries are encoded through a generic Avro value tree on every written row [hudi-rs]

["yihua (via GitHub)" <gi...@apache.org>]

yihua opened a new issue, #872:
URL: https://github.com/apache/hudi-rs/issues/872

   ### Feature Description
   
   Every commit on a table with the record-level index writes one index entry per written row, and each entry is encoded by building an `apache_avro::Value` tree and calling `to_avro_datum` against the full `HoodieMetadataRecord` schema. That per-row tree, its allocations and the schema walk dominate the write: appending 1M rows takes about 21.6 s with the record index on versus about 1.6 s with it off.
   
   Proposed: encode the fixed `HoodieMetadataRecord` shape for a record index entry straight into Avro binary, and hold it byte for byte to the generic encoding in tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Record index entries are encoded through a generic Avro value tree on every written row [hudi-rs]

["yihua (via GitHub)" <gi...@apache.org>]

yihua closed issue #872: Record index entries are encoded through a generic Avro value tree on every written row
URL: https://github.com/apache/hudi-rs/issues/872


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org