zip from zip with mappers

[Patrick Martin <an...@gmail.com>]

Hello,

I have a situation where I need to create an archive (zip) from an other
one, but transforming some path inside. I want to avoid using a temporary
folder structure as the input files contains a huge amount of entries
(about 125000...) and this generates quite a lot of I/Os.

I was thinking of using <zip> and <zipfileset>. However, as <zip> does not
support <mapper>, it is not possible directly.

So I thought of using the <mappedresources> resource collection:
<zip destfile="dest.zip">
<mappedresources enablemultiplemappings="true">
<zipfileset src="src.zip" includes="**" />
<firstmatchmapper> <mapper type="glob" from="fromfolder\*" to="tofolder\*"
/> <identitymapper /> </firstmatchmapper>
</mappedresources>
</zip>

The mapping is done quite fast but then zip is reeaaally slow (can be seen
when running in verbose mode). My guess is that Ant will look for each
resource individually and thus scan the source zip each time.

Is there any other possibility to achieve zip to zip transfer with mapping
/ transformation (and filtering with include/exclude patterns) ?
Or is there any way to add <mapper> support in <zip> ?

Thank you,

Patrick

Re: zip from zip with mappers

[Stefan Bodewig <bo...@apache.org>]

On 2012-11-20, Patrick Martin wrote:

> Is there any way this can be available, in the compress antlib maybe?

The main problem is the way zipfileset and zipentry work not the zip
task - and the compress Antlib doesn't address this, either.

> Is the addition of <mapper> support directly in <zip> not possible either
> (so we could make use of the zipfileset nested directly, but mapping zip
> entry names at the last step)?

At the surface this doesn't look too hard but it becomes very difficult
once you realize that most methods you have to touch are protected and
there are subclasses overriding them - so you can't change their
signatures.  This (the difficulty of modifying the zip task) is one of
the reasons the compress Antlib was started.

It would be possible to aff mapper support to the compressing tasks in
the compress Antlib but then again the Zip task over there doesn't
contain any special handling for zipfilesets and would be as slow as
using mappedresources.

If you wanted to tackle the problem by patching Ant, working on
ZipFileSet and/or ArchiveFileSet as well as the ArchiveEntry subclasses
is the way to go.  The rationale for the current behavior is that the
code is very conservative: when ZipFileSet hands out the ZipResources it
cannot know when the opened archive is no longer needed so in order to
avoid leaks each method that needs something from the archive is
responsible for opening and closing it.  Any solution that kept the
archive open must ensure the archive is eventually closed - and we can't
rely on the garbage collector here.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org

Re: zip from zip with mappers

[Patrick Martin <an...@gmail.com>]

Hello,

Thank you for your reply,

Is there any way this can be available, in the compress antlib maybe?
Is the addition of <mapper> support directly in <zip> not possible either
(so we could make use of the zipfileset nested directly, but mapping zip
entry names at the last step)?

Thank you again,

Patrick


On Tue, Nov 20, 2012 at 6:05 AM, Stefan Bodewig <bo...@apache.org> wrote:

> On 2012-11-19, Patrick Martin wrote:
>
> > I have a situation where I need to create an archive (zip) from an other
> > one, but transforming some path inside. I want to avoid using a temporary
> > folder structure as the input files contains a huge amount of entries
> > (about 125000...) and this generates quite a lot of I/Os.
>
> > I was thinking of using <zip> and <zipfileset>. However, as <zip> does
> not
> > support <mapper>, it is not possible directly.
>
> > So I thought of using the <mappedresources> resource collection:
> > <zip destfile="dest.zip">
> > <mappedresources enablemultiplemappings="true">
> > <zipfileset src="src.zip" includes="**" />
> > <firstmatchmapper> <mapper type="glob" from="fromfolder\*"
> to="tofolder\*"
> > /> <identitymapper /> </firstmatchmapper>
> > </mappedresources>
> > </zip>
>
> > The mapping is done quite fast but then zip is reeaaally slow (can be
> seen
> > when running in verbose mode).
>
> Yes, this is a known problem with the way Ant deals with zipfileset.
>
> > My guess is that Ant will look for each resource individually and thus
> > scan the source zip each time.
>
> It's even worse.  The ZIP file will be reopened and the central
> directory re-parsed every time an entry is read.  This is avoided by
> special code inside the zip task when the zipfileset is nested directly
> into the task.
>
> > Is there any other possibility to achieve zip to zip transfer with
> mapping
> > / transformation (and filtering with include/exclude patterns) ?
> > Or is there any way to add <mapper> support in <zip> ?
>
> Filtering should work by using zipfileset directly.  As for mapping, I'm
> afraid there is no real solution - apart from coding the ZIP creation
> directly in Java.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
>
>

Re: zip from zip with mappers

[Stefan Bodewig <bo...@apache.org>]

On 2012-11-19, Patrick Martin wrote:

> I have a situation where I need to create an archive (zip) from an other
> one, but transforming some path inside. I want to avoid using a temporary
> folder structure as the input files contains a huge amount of entries
> (about 125000...) and this generates quite a lot of I/Os.

> I was thinking of using <zip> and <zipfileset>. However, as <zip> does not
> support <mapper>, it is not possible directly.

> So I thought of using the <mappedresources> resource collection:
> <zip destfile="dest.zip">
> <mappedresources enablemultiplemappings="true">
> <zipfileset src="src.zip" includes="**" />
> <firstmatchmapper> <mapper type="glob" from="fromfolder\*" to="tofolder\*"
> /> <identitymapper /> </firstmatchmapper>
> </mappedresources>
> </zip>

> The mapping is done quite fast but then zip is reeaaally slow (can be seen
> when running in verbose mode).

Yes, this is a known problem with the way Ant deals with zipfileset.

> My guess is that Ant will look for each resource individually and thus
> scan the source zip each time.

It's even worse.  The ZIP file will be reopened and the central
directory re-parsed every time an entry is read.  This is avoided by
special code inside the zip task when the zipfileset is nested directly
into the task.

> Is there any other possibility to achieve zip to zip transfer with mapping
> / transformation (and filtering with include/exclude patterns) ?
> Or is there any way to add <mapper> support in <zip> ?

Filtering should work by using zipfileset directly.  As for mapping, I'm
afraid there is no real solution - apart from coding the ZIP creation
directly in Java.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org