You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Maheshinder Goyal (Jira)" <ji...@apache.org> on 2023/10/29 23:09:00 UTC
[jira] [Comment Edited] (COMPRESS-649) Performance Degradation in LZ4 compression between 1.21 and 1.22 and later versions
[ https://issues.apache.org/jira/browse/COMPRESS-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17780781#comment-17780781 ]
Maheshinder Goyal edited comment on COMPRESS-649 at 10/29/23 11:08 PM:
-----------------------------------------------------------------------
HI [~ggregory]
Thanks again for resolving the issue.
For sake of curiosity, I have few followup questions:
# Is it feasible to share what was the reason for degradation? Asking this because if it is something generic, then myself and team can also learn as a good design and practice for future in our development as well. If you think it was too specific and not worth sharing, then please ignore.
# Not sure how this degradation got ignored on your side in terms of regression scenarios. Have we taken care of it in future by adding some sort of unit-test or integration test?
# When shall I expect tentatively, official release of 1.25.0 to have this issue addressed?
Thanks & Regards,
Maheshinder Goyal
was (Author: JIRAUSER302790):
HI Gary,
Thanks again for resolving the issue.
For sake of curiosity, I have few followup questions:
# Is it feasible to share what was the reason for degradation? Asking this because if it is something generic, then myself and team can also learn as a good design and practice for future in our development as well. If you think it was too specific and not worth sharing, then please ignore.
# Not sure how this degradation got ignored on your side in terms of regression scenarios. Have we taken care of it in future by adding some sort of unit-test or integration test?
# When shall I expect tentatively, official release of 1.25.0 to have this issue addressed?
Thanks & Regards,
Maheshinder Goyal
> Performance Degradation in LZ4 compression between 1.21 and 1.22 and later versions
> -----------------------------------------------------------------------------------
>
> Key: COMPRESS-649
> URL: https://issues.apache.org/jira/browse/COMPRESS-649
> Project: Commons Compress
> Issue Type: Bug
> Components: Compressors
> Affects Versions: 1.22, 1.23.0, 1.24.0
> Reporter: Maheshinder Goyal
> Priority: Major
> Fix For: 1.25.0
>
> Attachments: CompressionDegradationTest.java, some-900kb-text.txt
>
>
> Hi Team,
>
> We use LZ4 compression in our project to compress data under 1MB. We generally deal with String to String compression.
> We typically compress around 50 to 100 strings in a loop as part of a function in one call.
> When we got new Apache-Commons-compress version of 1.24 we started noticing a degradation in performance where our workflow started taking more than double time in seconds than it used to take with 1.21 version.
> We reverted the change and went back to 1.21 and performance returned back to good.
> Now, we have reproduced the issue in a small standalone java unit-test where we have noticed 2x performance degradation on a small example.
>
> In the following example, we have used a text-file with name "some-900kb-text.txt" which can be any random text file of 900KB size.
>
> You will notice that b/w 1.24 and 1.21 , the following program would take 8 to 9 seconds with 1.24 version and around 3 seconds with 1.21 version.
>
> If you increase the number of loops, the performance will degrade further.
> ###################################################
> {code:java}
> import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.io.RandomAccessFile;
> import java.nio.ByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.charset.StandardCharsets;
> import java.util.Base64;
> public class CompressionDegradationTest {
> public static void main(String[] args) throws Exception {
> RandomAccessFile aFile = new RandomAccessFile("some-900kb-text.txt", "r");
> FileChannel inChannel = aFile.getChannel();
> long fileSize = inChannel.size();
> ByteBuffer buffer = ByteBuffer.allocate((int) fileSize);
> inChannel.read(buffer);
> buffer.flip();
> String rawPlan = new String(buffer.array(), StandardCharsets.UTF_8);
> long start = System.currentTimeMillis();
> for (int i = 0; i < 80; i++) {
> String compressed = compress(rawPlan);
> }
> long end = System.currentTimeMillis();
> float sec = (end - start) / 1000F;
> System.out.println(sec + " seconds");
> }
> private static String compress(final String value) throws IOException {
> ByteArrayOutputStream byteStream = new ByteArrayOutputStream(value.length());
> FramedLZ4CompressorOutputStream compress = new FramedLZ4CompressorOutputStream(byteStream);
> String compressedValue = null;
> try {
> compress.write(value.getBytes(StandardCharsets.UTF_8));
> compress.finish();
> compressedValue = Base64.getEncoder().encodeToString(byteStream.toByteArray());
> } finally {
> compress.close();
> byteStream.close();
> }
> return compressedValue;
> }
> };
> {code}
> ########################################################
--
This message was sent by Atlassian Jira
(v8.20.10#820010)