You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Tomasz Bigosiński <tb...@sii.pl.INVALID> on 2023/01/12 08:28:54 UTC

[compress] ParallelScatterZipCreator writes to user tmp folder

Hi,

I have a question on using your Apache Commons Compress library. I would like to use it for my project and compress many files from some folder into one zip archive with multithreading.

There is a sample code piece here Commons Compress - Commons Compress User Guide (apache.org)<https://commons.apache.org/proper/commons-compress/examples.html> ('Creating a zip file with multiple threads' section) and an article here java - How to implement Parallel Zip Creation with ScatterZipOutputStream with Zip64 Support? - Stack Overflow<https://stackoverflow.com/questions/54624695/how-to-implement-parallel-zip-creation-with-scatterzipoutputstream-with-zip64-su>.

Eventually I managed to have this working inside my project's code, but I noticed that when writting into zip archive it first creates chunks in /tmp folder, which is not the best solution for me - I have rather big files to serve (hundreds of GB each) and it makes no sense for me to have /tmp folder too big on an everyday basis.

So is there a way to force ParallelScatterZipCreator class to write temporary files to some other location, e.g. the same as the final archive?

Regards,
Tomasz Bigosinski

Re: [compress] ParallelScatterZipCreator writes to user tmp folder

Posted by "Gary D. Gregory" <gg...@apache.org>.
Hello Tomasz,

Sounds reasonable, please see git master for the class DefaultBackingStoreSupplier (or a SNAPSHOT build in the Apache SNAPSHOT Maven repository: https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-compress/1.23-SNAPSHOT/), and instead of:

final ParallelScatterZipCreator zipCreator = new ParallelScatterZipCreator();

Use:
        final Path dir = Paths.get("target/custom-temp-dir");
        Files.createDirectories(dir);
        final ParallelScatterZipCreator zipCreator = new ParallelScatterZipCreator(
                Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()),
                new DefaultBackingStoreSupplier(dir));

FWIW, I do not think we should add new constructors.

Gary

On 2023/01/12 08:28:54 Tomasz Bigosiński wrote:
> Hi,
> 
> I have a question on using your Apache Commons Compress library. I would like to use it for my project and compress many files from some folder into one zip archive with multithreading.
> 
> There is a sample code piece here Commons Compress - Commons Compress User Guide (apache.org)<https://commons.apache.org/proper/commons-compress/examples.html> ('Creating a zip file with multiple threads' section) and an article here java - How to implement Parallel Zip Creation with ScatterZipOutputStream with Zip64 Support? - Stack Overflow<https://stackoverflow.com/questions/54624695/how-to-implement-parallel-zip-creation-with-scatterzipoutputstream-with-zip64-su>.
> 
> Eventually I managed to have this working inside my project's code, but I noticed that when writting into zip archive it first creates chunks in /tmp folder, which is not the best solution for me - I have rather big files to serve (hundreds of GB each) and it makes no sense for me to have /tmp folder too big on an everyday basis.
> 
> So is there a way to force ParallelScatterZipCreator class to write temporary files to some other location, e.g. the same as the final archive?
> 
> Regards,
> Tomasz Bigosinski
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org