You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Kristian Rosenvold <kr...@apache.org> on 2014/12/15 08:49:45 UTC
[compress] Changes to allow multithreaded creation of zip files
I just thought I'd let you know I'm working on changes to allow
multiple threads to output to different ZipArchiveOutputStream
instances (backed by commons-io DeferredFileOutputStream or similar)
and then stitch the results back together as a single Zip file. I'm
currently just researching the scope of the required changes (which
seems to be fairly small) and working on test cases. I expect to be
working with this for a few weeks, my code can be seen at
https://github.com/krosenvold/commons-compress
(And I'll be implementing this in maven once it's good).
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Changes to allow multithreaded creation of zip files
Posted by Benedikt Ritter <br...@apache.org>.
Hello Kristian,
2014-12-15 8:49 GMT+01:00 Kristian Rosenvold <kr...@apache.org>:
>
> I just thought I'd let you know I'm working on changes to allow
> multiple threads to output to different ZipArchiveOutputStream
> instances (backed by commons-io DeferredFileOutputStream or similar)
> and then stitch the results back together as a single Zip file. I'm
> currently just researching the scope of the required changes (which
> seems to be fairly small) and working on test cases. I expect to be
> working with this for a few weeks, my code can be seen at
> https://github.com/krosenvold/commons-compress
>
> (And I'll be implementing this in maven once it's good).
>
Very nice! Can you please also file a jira for your changes?
TIA!
Benedikt
>
> Kristian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
--
http://people.apache.org/~britter/
http://www.systemoutprintln.de/
http://twitter.com/BenediktRitter
http://github.com/britter
Re: [compress] Changes to allow multithreaded creation of zip files
Posted by Kristian Rosenvold <kr...@apache.org>.
I just landed two commits for COMMONS-295 and COMMONS-296, which is
the groundwork for proper parallel support. The icing on the cake (the
high-level api with "nThreads" configuration and single point of
entry) is still work-in-progress and can be seen at
https://github.com/krosenvold/commons-compress/tree/concurrentSupport
It's still early days for the "high level service"; it needs to do
lots of funky stuff like balance large files first and use work
stealing towards the end. Currently I can easily make this use 850%
CPU on my 6 core cpu.
The "cool" stuff is still in github, but it's a lot easier for
"anyone" to build and play around with now.
Kristian
2014-12-18 9:29 GMT+01:00 Kristian Rosenvold <kr...@apache.org>:
> +1 for moving to git btw :)
>
> Kristian
>
> 2014-12-15 11:38 GMT+01:00 Emmanuel Bourg <eb...@apache.org>:
>> Le 15/12/2014 11:36, Stefan Bodewig a écrit :
>>
>>> [as an aside, maybe we should think about moving Compress to git]
>>
>> +1
>>
>> Emmanuel
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Changes to allow multithreaded creation of zip files
Posted by Kristian Rosenvold <kr...@apache.org>.
+1 for moving to git btw :)
Kristian
2014-12-15 11:38 GMT+01:00 Emmanuel Bourg <eb...@apache.org>:
> Le 15/12/2014 11:36, Stefan Bodewig a écrit :
>
>> [as an aside, maybe we should think about moving Compress to git]
>
> +1
>
> Emmanuel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Changes to allow multithreaded creation of zip files
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 15/12/2014 11:36, Stefan Bodewig a écrit :
> [as an aside, maybe we should think about moving Compress to git]
+1
Emmanuel
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Changes to allow multithreaded creation of zip files
Posted by Stefan Bodewig <bo...@apache.org>.
On 2014-12-15, Kristian Rosenvold wrote:
> I just thought I'd let you know I'm working on changes to allow
> multiple threads to output to different ZipArchiveOutputStream
> instances (backed by commons-io DeferredFileOutputStream or similar)
> and then stitch the results back together as a single Zip file. I'm
> currently just researching the scope of the required changes (which
> seems to be fairly small) and working on test cases.
Great, I'm looking forward to them.
> I expect to be working with this for a few weeks, my code can be seen
> at https://github.com/krosenvold/commons-compress
[as an aside, maybe we should think about moving Compress to git]
> (And I'll be implementing this in maven once it's good).
AFAIU Damjan would like to see a new Compress release to move Imaging
forward; I'd volunteer to manage a new release after we've integrated
and tested your changes.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org