Using Ivy with very large input content

[Pa...@nokia.com]

Hello,

This is a question about scalability and also how well Ivy is suited to
non-Java projects. We use Ant for large builds involving Gbs of data.
I'm interested in how Ivy could be applied. Typically a build will have
a number of inputs which include unzipping several Gbs of zips and also
copying several more Gb of source files. While Ant is very useful for
managing the build process, trials found that it does not currently
scale to large copies and unzips, so we have some other custom scripting
to handle the preparation.

I could imagine using Ivy to define the dependencies between the needed
inputs. 

Would this make sense? Has Ivy been used for non-Java builds much?

The notion of the cache may not make sense for some of these large
inputs. Is it enough to use the useOrigin attribute? 

How does Ivy handle the copying of data around, and is this pluggable?
Ideal option would be that the operations to copy stuff could be
customised to use the fastest copy operations on the platform (Win,
Linux), which is kinda what we do now. Or maybe even an output file that
simply listed the required copys after Ivy

All suggestions and input much appreciated!

thanks

paul

Re: Using Ivy with very large input content

[Xavier Hanin <xa...@gmail.com>]

On Jan 11, 2008 3:47 AM, <Pa...@nokia.com> wrote:

> Hello,
>
> This is a question about scalability and also how well Ivy is suited to
> non-Java projects. We use Ant for large builds involving Gbs of data.
> I'm interested in how Ivy could be applied. Typically a build will have
> a number of inputs which include unzipping several Gbs of zips and also
> copying several more Gb of source files. While Ant is very useful for
> managing the build process, trials found that it does not currently
> scale to large copies and unzips, so we have some other custom scripting
> to handle the preparation.
>
> I could imagine using Ivy to define the dependencies between the needed
> inputs.
>
> Would this make sense?

It's difficult to tell without more details on your needs. IMO Ivy shines if
you need to handle transitive dependencies, or just want a good abstraction
between a module repository and your build.

> Has Ivy been used for non-Java builds much?

It's very difficult to say, I've seen many people speaking about non java
builds on the lists and previous forums. But does it only mean that using
Ivy for non java build is more difficult, so people always ask questions.
And did they actually end up using Ivy for their build, I don't really know.
What I can say is that Ivy has not been designed to handle java dependencies
only, and we are always open to requirements from non java land.


>
> The notion of the cache may not make sense for some of these large
> inputs. Is it enough to use the useOrigin attribute?

It is. And with current work being done on cache management, Ivy 2.0 will
provide even easier and powerful control over caching.

>
> How does Ivy handle the copying of data around, and is this pluggable?

Ivy uses java copy, so it's basically the same as Ant. I've had very good
results with this kind of copy, but it's probably not as good as native copy
as you point out. Some data copy is pluggable (the copy from the repository
to the cache, which you'd like to avoid), but not the copy from the cache to
another local destination using retrieve. The copy handling is concentrated
in one class only though (FileUtil), so patching this class (and even
providing a patch to support pluggability here) is possible.

But maybe you don't need this copy, if you avoid the cache Ivy can help you
locate the files you depend on where they are directly. Or maybe you could
consider this copy as "caching": with current trunk version of Ivy you
already have pretty good control over what caching means (basically it must
ensure that files are available on the filesystem, that's all), and where to
put the "cached" files. And since copy to cache is pluggable, you could
leverage this instead of a retrieve.

HTH,

Xavier


> Ideal option would be that the operations to copy stuff could be
> customised to use the fastest copy operations on the platform (Win,
> Linux), which is kinda what we do now. Or maybe even an output file that
> simply listed the required copys after Ivy
>
> All suggestions and input much appreciated!
>
> thanks
>
> paul
>
>


-- 
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/