You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Gary D. Gregory (Jira)" <ji...@apache.org> on 2023/10/26 01:43:00 UTC
[jira] [Updated] (COMPRESS-649) Performance Degradation in LZ4 compression between 1.21 and 1.22 and later versions
[ https://issues.apache.org/jira/browse/COMPRESS-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary D. Gregory updated COMPRESS-649:
-------------------------------------
Summary: Performance Degradation in LZ4 compression between 1.21 and 1.22 and later versions (was: Performance Degradation in LZ4 compression between 1.21 and 1.24 versions)
> Performance Degradation in LZ4 compression between 1.21 and 1.22 and later versions
> -----------------------------------------------------------------------------------
>
> Key: COMPRESS-649
> URL: https://issues.apache.org/jira/browse/COMPRESS-649
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
> Affects Versions: 1.22, 1.23.0, 1.24.0
> Reporter: Maheshinder Goyal
> Priority: Major
> Fix For: 1.21
>
> Attachments: CompressionDegradationTest.java
>
>
> Hi Team,
>
> We use LZ4 compression in our project to compress data under 1MB. We generally deal with String to String compression.
> We typically compress around 50 to 100 strings in a loop as part of a function in one call.
> When we got new Apache-Commons-compress version of 1.24 we started noticing a degradation in performance where our workflow started taking more than double time in seconds than it used to take with 1.21 version.
> We reverted the change and went back to 1.21 and performance returned back to good.
> Now, we have reproduced the issue in a small standalone java unit-test where we have noticed 2x performance degradation on a small example.
>
> In the following example, we have used a text-file with name "some-900kb-text.txt" which can be any random text file of 900KB size.
>
> You will notice that b/w 1.24 and 1.21 , the following program would take 8 to 9 seconds with 1.24 version and around 3 seconds with 1.21 version.
>
> If you increase the number of loops, the performance will degrade further.
> ###################################################
> {code:java}
> import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.io.RandomAccessFile;
> import java.nio.ByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.charset.StandardCharsets;
> import java.util.Base64;
> public class CompressionDegradationTest {
> public static void main(String[] args) throws Exception {
> RandomAccessFile aFile = new RandomAccessFile("some-900kb-text.txt", "r");
> FileChannel inChannel = aFile.getChannel();
> long fileSize = inChannel.size();
> ByteBuffer buffer = ByteBuffer.allocate((int) fileSize);
> inChannel.read(buffer);
> buffer.flip();
> String rawPlan = new String(buffer.array(), StandardCharsets.UTF_8);
> long start = System.currentTimeMillis();
> for (int i = 0; i < 80; i++) {
> String compressed = compress(rawPlan);
> }
> long end = System.currentTimeMillis();
> float sec = (end - start) / 1000F;
> System.out.println(sec + " seconds");
> }
> private static String compress(final String value) throws IOException {
> ByteArrayOutputStream byteStream = new ByteArrayOutputStream(value.length());
> FramedLZ4CompressorOutputStream compress = new FramedLZ4CompressorOutputStream(byteStream);
> String compressedValue = null;
> try {
> compress.write(value.getBytes(StandardCharsets.UTF_8));
> compress.finish();
> compressedValue = Base64.getEncoder().encodeToString(byteStream.toByteArray());
> } finally {
> compress.close();
> byteStream.close();
> }
> return compressedValue;
> }
> };
> {code}
> ########################################################
--
This message was sent by Atlassian Jira
(v8.20.10#820010)