[jira] [Created] (LUCENE-10112) Improve LZ4 Compression performance with direct primitive read/writes

["Tim Brooks (Jira)" <ji...@apache.org>]

Tim Brooks created LUCENE-10112:
-----------------------------------

             Summary: Improve LZ4 Compression performance with direct primitive read/writes
                 Key: LUCENE-10112
                 URL: https://issues.apache.org/jira/browse/LUCENE-10112
             Project: Lucene - Core
          Issue Type: Improvement
            Reporter: Tim Brooks


*Summary*

Java9 introduced VarHandles as a tool to quickly read and write primitive types directly to byte arrays without bound checks. The LZ4 compressor class must consistently read ints from a byte array to analyze matches. The performance can be improved by reading these using a VarHandle.

Additionally, the LZ4 compressor/decompressor methods currently individually read/write the bytes for LE shorts. Lucene's DataOutput/DataInput abstractions already have dedicated methods for reading/writing LE shorts. These methods are selectively optimized in certain implementations and will provide superior performance than individual byte reads.

*Concerns*

The DataOutput/DataInput readShort() and writeShort() methods do not call out that they are LE. It just looks to me that the DataOutput/DataInput are LE? Since this particular change does not appear to provide significant performance wins, maybe the patch is better leaving the explicit individual byte reads?

Additionally, this patch changes read ints to read them in the platform native order which should be fine since it is just matching bytes. But I can change it to only read in the order the previous version did.

*Benchmarks*

I created JMH benchmarks which compresses 1MB of highly compressible JSON observability data. And compresses it 64KB at a time. In order to simulate the "short" changes, I use a forked version `ByteArrayDataOutput` which writes shorts using a VarHandle (to simulate fast writes that the ByteBuffer versions would get.) I also ran a benchmark without the short changes, just the reading ints using a VarHandle.

 

 
{noformat}
Benchmark                                          Mode  Cnt    Score   Error  Units
MyBenchmark.testCompressLuceneLZ4                 thrpt    9  712.430 ± 3.616  ops/s
MyBenchmark.testCompressLuceneLZ4Forked           thrpt    9  945.380 ± 4.776  ops/s
MyBenchmark.testCompressLuceneLZ4ForkedNoShort    thrpt    9  940.812 ± 3.868  ops/s
MyBenchmark.testCompressLuceneLZ4HC               thrpt    9  147.432 ± 4.730  ops/s
MyBenchmark.testCompressLuceneLZ4HCForked         thrpt    9  183.954 ± 2.534  ops/s
MyBenchmark.testCompressLuceneLZ4HCForkedNoShort  thrpt    9  188.065 ± 0.727  ops/s{noformat}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org