[compress] LZ4 compression taking excessively long

[Mark Raynsford <or...@io7m.com.INVALID>]

Hello!

I've run into an issue when trying to use the
FramedLZ4CompressorOutputStream class in commons-compress 1.21.

I've put together a small repro case here:

  https://github.com/io7m/commons-compress-20220114

If you run the BadCompress class, it will attempt to open
and compress the given 2097152 byte file. What seems to happen is that
the code goes into an excessively long loop after the finish() method
is called.

The standard lz4 command-line tool doesn't have any problems with the
input:

$ time lz4 bad-compress.bin 
Compressed filename will be : bad-compress.bin.lz4 
Compressed 2097152 bytes into 589156 bytes ==> 28.09%
real    0m0.005s
user    0m0.003s
sys     0m0.003s

I've yet to see the BadCompress example actually finish executing...

Is there something I'm doing wrong? I've tried, for example,
submitting 2097152 bytes as chunks of 4096 bytes, but it makes
no difference.

-- 
Mark Raynsford | https://www.io7m.com

Re: [compress] LZ4 compression taking excessively long

["Bruno P. Kinoshita" <br...@yahoo.com.br.INVALID>]

 Hello,
I reproduced your test case. It completed when running without the debugger:


But with the debugger it didn't finish in a couple minutes so I killed it. Then going through the code, it really takes a long time in the finish() method. There are a few loops that could be the reason for the problem (e.g. https://github.com/apache/commons-compress/blob/6aacb2953d03b9423057f176e6d4cbe6dca65b3d/src/main/java/org/apache/commons/compress/compressors/lz77support/LZ77Compressor.java#L421) and also some LZ77 parameters that could speed up the compression.

I am not really that familiar with the [compress] code base. But if you create a JIRA issue, someone else might be able to help investigating this issue. I will try to take a look if someday I find time to work on [compress]. Since we have another library to compare, it shouldn't be too hard to figure out what's wrong, and compare the performance of the two libraries.

Thanks!
-Bruno


    On Saturday, 15 January 2022, 09:50:09 am NZDT, Mark Raynsford <or...@io7m.com.invalid> wrote:  
 
 Hello!

I've run into an issue when trying to use the
FramedLZ4CompressorOutputStream class in commons-compress 1.21.

I've put together a small repro case here:

  https://github.com/io7m/commons-compress-20220114

If you run the BadCompress class, it will attempt to open
and compress the given 2097152 byte file. What seems to happen is that
the code goes into an excessively long loop after the finish() method
is called.

The standard lz4 command-line tool doesn't have any problems with the
input:

$ time lz4 bad-compress.bin 
Compressed filename will be : bad-compress.bin.lz4 
Compressed 2097152 bytes into 589156 bytes ==> 28.09%
real    0m0.005s
user    0m0.003s
sys    0m0.003s

I've yet to see the BadCompress example actually finish executing...

Is there something I'm doing wrong? I've tried, for example,
submitting 2097152 bytes as chunks of 4096 bytes, but it makes
no difference.

-- 
Mark Raynsford | https://www.io7m.com