You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ant.apache.org by Antti Luoma <an...@gmail.com> on 2008/01/24 10:53:13 UTC

Syncing very large directories

Hi,

I am using ant 1.7.0 <sync> task to sync 2 very large directories. It
currently takes too much time to do this (to be usable).
Here is the target that I run.

<target name="sync.dir" description="runs sync between ${src} and ${dest}" >
    <sync todir="${dest}" overwrite="false" verbose="true">
      <fileset dir="${src}"/>
    </sync>
  </target>

I invoked it whit build metrics listener and the output that I (finally) got
was:

     [sync] Removed 1 dangling directory from C:\Work\
     [sync] Removed 12 dangling files from C:\Work\
     [sync] Removed 141 empty directories from C:\Work\

BUILD SUCCESSFUL
Total time: 944 minutes 28 seconds
BUILD METRICS:
Local Time, Child Time, Invocation Count, Type, Name, Location
56667816, 0, 1, TASK, sync, C:\Work\build.xml:36:
3237, 56667816, 1, BUILD, build, n/a:
0, 56667816, 1, TARGET, sync.sdk, C:\Work\build.xml:34:

Listener that I used was: -listener net.java.antutility.BuildMetricsL


Finally the question is what is the fastest way to sync two large
directories? This obivously isn't. I was thinking that maybe using resource
collections and diffrence sets would be better.. Any ideas.

Also in this case it is possible to first store the index of files into an
object in disk and compare that to destination directory. This could be
faster because it wouldn't have to scan source dir at all..

-Antti-

Re: Syncing very large directories

Posted by Antti Luoma <an...@gmail.com>.
I agree that something isn't scaling too well. Sync works ok with
directories < 1 GB ( ~17 000 files, 1300 folders) less than a minute..

There are ~ 500 000 (half a million) files, and ~74 000 folders.

When I browsed the sync / copy code I found that there was a lot of
synchronized code blocks. I doubt that it works much more faster with
paraller tasks (not tried, yet..).

-- 
-Antti-

2008/1/25, Dominique Devienne <dd...@gmail.com>:
>
> On Jan 25, 2008 1:02 AM, Antti Luoma <an...@gmail.com> wrote:
> > BTW, very large in this case means ~17 GB :)
>
> Well, transferring 17GB @ 100Mb/s takes in theory 1350 seconds (less
> than 23 minutes).
>
> You indicated a total time for the sync of 56667816 seconds, which is
> about 300 bytes / seconds.
>
> Even accounting for non-ideal network conditions and directory
> traversal, there's still a huge gap between 100Mb/s and 2.4Kb/s
> (40,000x different in speed...).
>
> Something doesn't quite compute here ;-) How many files and dirs do
> you have in these 17GB? It's possible some of the data-structure used
> in sync don't scale well for a very large number of files...
>
> Assuming the dir you sync has a few well known top level dirs, you
> could also sync those instead of the "root" sync dir, and put several
> sync in <parallel>. --DD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
>
>

Re: Syncing very large directories

Posted by Dominique Devienne <dd...@gmail.com>.
On Jan 25, 2008 1:02 AM, Antti Luoma <an...@gmail.com> wrote:
> BTW, very large in this case means ~17 GB :)

Well, transferring 17GB @ 100Mb/s takes in theory 1350 seconds (less
than 23 minutes).

You indicated a total time for the sync of 56667816 seconds, which is
about 300 bytes / seconds.

Even accounting for non-ideal network conditions and directory
traversal, there's still a huge gap between 100Mb/s and 2.4Kb/s
(40,000x different in speed...).

Something doesn't quite compute here ;-) How many files and dirs do
you have in these 17GB? It's possible some of the data-structure used
in sync don't scale well for a very large number of files...

Assuming the dir you sync has a few well known top level dirs, you
could also sync those instead of the "root" sync dir, and put several
sync in <parallel>. --DD

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org


Re: Syncing very large directories

Posted by Antti Luoma <an...@gmail.com>.
BTW, very large in this case means ~17 GB :)
-- 
-Antti-

2008/1/25, Antti Luoma <an...@gmail.com>:
>
> Hi,
>
> I did take a look from sources and noticed the basic java File API, I
> think there might be some optimization (like getting rid of unnecessary
> vector creation). Also caching src folder contents (is maybe storing
> contenst to outputstream directly and reading from there) might improve it?
> This of course is if you sync SAME directories..  Or maybe somebody has an
> even better idea?
>
> I'm stuck with windows platform (due to sdk that I'm using).. But i'll
> think I give rsync a try (even though used linux for years this is something
> that I haven't worked with before). It might even work better on my case
> (client / server model..). Only side effect it that i'll need another
> prorgram and more setup to make this system rock (not so easy to transfer to
> another machine, that would have been the best part with using build in ant
> tasks).
>
> But thanks!
>
>
> --
> -Antti-
>
> 2008/1/24, Dominique Devienne <dd...@gmail.com>:
> >
> > On Jan 24, 2008 3:53 AM, Antti Luoma <an...@gmail.com> wrote:
> > > Hi,
> > >
> > > I am using ant 1.7.0 <sync> task to sync 2 very large directories. It
> > > currently takes too much time to do this (to be usable).
> >
> > Sorry Antti, but there's no free lunch. Either you want to sync, in
> > which case you need to scan both the source and the destination
> > folders, or you don't. The whole point of sync is that things can
> > change in either folders, so you must scan them to know what.
> >
> > <sync> uses a simple minded algo, using existing Java File APIs, which
> > aren't that fast, especially with remote file systems. I don't see how
> > to speed it up without compromising the goal of syncing things.
> >
> > rsync OTOH is a much smarter approach, but relies on a client-server
> > approach where both the source and destination must be rsync-enabled
> > to be really fast (AFAIK), and not very win32-friendly (I could not
> > make it work with UNC paths for examples).
> >
> > 15 hours is a long time. That must be one mighty big folder... --DD
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> > For additional commands, e-mail: user-help@ant.apache.org
> >
> >
>
>
>

Re: Syncing very large directories

Posted by Antti Luoma <an...@gmail.com>.
Hi,

I did take a look from sources and noticed the basic java File API, I think
there might be some optimization (like getting rid of unnecessary vector
creation). Also caching src folder contents (is maybe storing contenst to
outputstream directly and reading from there) might improve it? This of
course is if you sync SAME directories..  Or maybe somebody has an even
better idea?

I'm stuck with windows platform (due to sdk that I'm using).. But i'll think
I give rsync a try (even though used linux for years this is something that
I haven't worked with before). It might even work better on my case (client
/ server model..). Only side effect it that i'll need another prorgram and
more setup to make this system rock (not so easy to transfer to another
machine, that would have been the best part with using build in ant tasks).

But thanks!


-- 
-Antti-

2008/1/24, Dominique Devienne <dd...@gmail.com>:
>
> On Jan 24, 2008 3:53 AM, Antti Luoma <an...@gmail.com> wrote:
> > Hi,
> >
> > I am using ant 1.7.0 <sync> task to sync 2 very large directories. It
> > currently takes too much time to do this (to be usable).
>
> Sorry Antti, but there's no free lunch. Either you want to sync, in
> which case you need to scan both the source and the destination
> folders, or you don't. The whole point of sync is that things can
> change in either folders, so you must scan them to know what.
>
> <sync> uses a simple minded algo, using existing Java File APIs, which
> aren't that fast, especially with remote file systems. I don't see how
> to speed it up without compromising the goal of syncing things.
>
> rsync OTOH is a much smarter approach, but relies on a client-server
> approach where both the source and destination must be rsync-enabled
> to be really fast (AFAIK), and not very win32-friendly (I could not
> make it work with UNC paths for examples).
>
> 15 hours is a long time. That must be one mighty big folder... --DD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
>
>

Re: Syncing very large directories

Posted by Dominique Devienne <dd...@gmail.com>.
On Jan 24, 2008 3:53 AM, Antti Luoma <an...@gmail.com> wrote:
> Hi,
>
> I am using ant 1.7.0 <sync> task to sync 2 very large directories. It
> currently takes too much time to do this (to be usable).

Sorry Antti, but there's no free lunch. Either you want to sync, in
which case you need to scan both the source and the destination
folders, or you don't. The whole point of sync is that things can
change in either folders, so you must scan them to know what.

<sync> uses a simple minded algo, using existing Java File APIs, which
aren't that fast, especially with remote file systems. I don't see how
to speed it up without compromising the goal of syncing things.

rsync OTOH is a much smarter approach, but relies on a client-server
approach where both the source and destination must be rsync-enabled
to be really fast (AFAIK), and not very win32-friendly (I could not
make it work with UNC paths for examples).

15 hours is a long time. That must be one mighty big folder... --DD

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org


Re: Syncing very large directories

Posted by Chuck Holzwarth <ch...@yahoo.com>.
Have you looked at rsync?

Antti Luoma <an...@gmail.com> wrote: Hi,

I am using ant 1.7.0  task to sync 2 very large directories. It
currently takes too much time to do this (to be usable).
Here is the target that I run.


    
      
    
  

I invoked it whit build metrics listener and the output that I (finally) got
was:

     [sync] Removed 1 dangling directory from C:\Work\
     [sync] Removed 12 dangling files from C:\Work\
     [sync] Removed 141 empty directories from C:\Work\

BUILD SUCCESSFUL
Total time: 944 minutes 28 seconds
BUILD METRICS:
Local Time, Child Time, Invocation Count, Type, Name, Location
56667816, 0, 1, TASK, sync, C:\Work\build.xml:36:
3237, 56667816, 1, BUILD, build, n/a:
0, 56667816, 1, TARGET, sync.sdk, C:\Work\build.xml:34:

Listener that I used was: -listener net.java.antutility.BuildMetricsL


Finally the question is what is the fastest way to sync two large
directories? This obivously isn't. I was thinking that maybe using resource
collections and diffrence sets would be better.. Any ideas.

Also in this case it is possible to first store the index of files into an
object in disk and compare that to destination directory. This could be
faster because it wouldn't have to scan source dir at all..

-Antti-



Thank you,
Chuck Holzwarth
(804) 403-3478 (home)
(540) 335-3171 (cell)
       
---------------------------------
Never miss a thing.   Make Yahoo your homepage.