[GitHub] [lucene] luyuncheng commented on pull request #987: LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on compressing data

[GitBox <gi...@apache.org>]

luyuncheng commented on PR #987:
URL: https://github.com/apache/lucene/pull/987#issuecomment-1200964321

   > Thanks for running these tests, let's remove `readNBytes` and switch to a shared byte[] that we call `DataInput#readBytes` on instead.
   
   @jpountz Thanks a lot for you reviewing for the code. at https://github.com/apache/lucene/pull/987/commits/c0d31d3134653bf6009b798925fa2350e5f7ec9c i removed readNBytes and using shared byte[] buffer in `DeflateWithPresetDictCompressionMode`. 
   
   I try to do more test about why `readNBytes` with a lot of memory copy, i think this is relevant to the:
   1. `DeflateWithPresetCompressingCodec` chunk size: `1 << 18`
   2. `LZ4WithPresetDictCompressionMode` NUM_SUB_BLOCKS: `10` 
   3. `ByteBuffersDataOutput` BITS_PER_BLOCK
   
   The `chunk size` is larger than `block size` which made `ByteBuffersDataInput` fragment always copy datas


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org