You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/07/28 16:23:33 UTC
[GitHub] [lucene] luyuncheng commented on pull request #987: LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on compressing data
luyuncheng commented on PR #987:
URL: https://github.com/apache/lucene/pull/987#issuecomment-1198369241
> Also I wonder if the `readNBytes` trick actually helps in practice, or if we end up crossing ByteBuffer boundaries most of the time in practice
@jpountz I have the same question that when crossing ByteBuffer would happened. So, i do a simple test:
As the result, i think `have a shared byte[]` is LGTM, i try to update it.
i rewrite the method with readNBytesWithoutCopy and readNBytesWithCopy as follows.
then i try to use arthas `monitor` with commands `monitor -c 10 org.apache.lucene.store.ByteBuffersDataInput readNBytesWith*` AND the running task is runStoredFieldsBenchmark with BEST_COMPRESSION:
it shows:
__#readNBytesWithoutCopy : #readNBytesWithCopy = 1 : 10__ every 10 seconds

codes:
```
public ByteBuffer readNBytes(int length) throws EOFException {
.....
if (block.remaining() >= length) {
this.pos += length;
return readNBytesWithoutCopy(block, blockOffset, length);
} else {
return readNBytesWithCopy(length);
}
}
private ByteBuffer readNBytesWithoutCopy(ByteBuffer block, int blockOffset, int length) {
return block.slice(blockOffset, length);
}
private ByteBuffer readNBytesWithCopy(int length) throws EOFException {
ByteBuffer copyBuffer = ByteBuffer.allocate(length);
readBytes(copyBuffer, length);
return copyBuffer.rewind().order(ByteOrder.LITTLE_ENDIAN);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org