You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ant.apache.org by Raja Nagendra Kumar <Na...@tejasoft.com> on 2009/09/13 12:56:15 UTC
DirectoryScanner cache to boost performance
Hi,
We have a custom tasks which use DirectoryScanner to eliminate the need for
developers trying to reference things by its relative path.
However, DirectoryScanner is so main tries with different exclude and
include pattern, which is resulting in perf issues.
In this context, is there a way to let the DirectoryScanner read the entire
file list only once in the entire ant session and every instance of
DirectoryScanner and its scans() would use such set to find the files
based on include and exclude patterns.
This approach would enable boost as finding the list of files available
search is avoided.
Regards,
Raja Nagendra Kumar,
C.T.O
www.tejasoft.com
-Web 2.0 Excellence
--
View this message in context: http://www.nabble.com/DirectoryScanner-cache-to-boost-performance-tp25422247p25422247.html
Sent from the Ant - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Gilles Scokart <gs...@gmail.com>.
FYI, Some months ago there has been some performance enhancements around the
directory scanner. I don't think this was already released in 1.7.1. You
can try to build the trunk if you want to benefits from those enhancements.
You can also search in the mail archive to know what has changed.
Gilles Scokart
2009/9/13 Raja Nagendra Kumar <Na...@tejasoft.com>
>
> Hi,
>
> We have a custom tasks which use DirectoryScanner to eliminate the need for
> developers trying to reference things by its relative path.
>
> However, DirectoryScanner is so main tries with different exclude and
> include pattern, which is resulting in perf issues.
>
> In this context, is there a way to let the DirectoryScanner read the entire
> file list only once in the entire ant session and every instance of
> DirectoryScanner and its scans() would use such set to find the files
> based on include and exclude patterns.
>
> This approach would enable boost as finding the list of files available
> search is avoided.
>
> Regards,
> Raja Nagendra Kumar,
> C.T.O
> www.tejasoft.com
> -Web 2.0 Excellence
> --
> View this message in context:
> http://www.nabble.com/DirectoryScanner-cache-to-boost-performance-tp25422247p25422247.html
> Sent from the Ant - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
>
>
Re: DirectoryScanner cache to boost performance
Posted by Raja Nagendra Kumar <Na...@tejasoft.com>.
Hi Stefan,
Thank you very much for this suggestion. We thought over this approach and
was able to replace the Directory Scanner with resource collection approach.
This resulted in 50% savings in time.. one which use to take 2 min per
device is now taking 1 min. Each of our build needs to run for 40 devices.
Entire build which use to take 80 min is now being completed with in 40 min.
Thank you for your expert suggestions.
Regards,
Nagendra
--
View this message in context: http://www.nabble.com/DirectoryScanner-cache-to-boost-performance-tp25422247p25624800.html
Sent from the Ant - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-09-15, Raja Nagendra Kumar <Na...@tejasoft.com> wrote:
> is there a way to use the include and exclude patterns as part of
> restrict.. just the name match is what we are looking far.
yes, use name selectors (and wrap them in "not"s for excludes) together
with correct "and"ing and "or"ing - this is what you already had in your
code IIRC.
There is no convenient API to do that, you'll have to chain them up
yourself. Sounds as if it might be worth investing some time in a
builder API for this, but since only the people who use Ant via Java
suffer from this (the XML API isn't less convenient than that of
<fileset>) it is unlikely to get written.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Raja Nagendra Kumar <Na...@tejasoft.com>.
yep.. missed to discover this..
Also, Stefan, is there a way to use the include and exclude patterns as part
of restrict..
just the name match is what we are looking far.
e.g patterns we wish to use are similar to the one we use in FileSet include
and excludes.
Regards,
Nagendra
--
View this message in context: http://www.nabble.com/DirectoryScanner-cache-to-boost-performance-tp25422247p25455827.html
Sent from the Ant - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-09-15, Raja Nagendra Kumar <Na...@tejasoft.com> wrote:
> Thank you very much for this info. I am trying to use this in our custom
> task some thing like this
> Union lUnion = (Union) TaskUtils.getCurrentProject().getReference(
> lRoot);
> Restrict lRestrict = new Restrict();
...
> However, I am unable to find a way to get the values matching such an
> expression. After doing the above, I am expecting some method in lRestrict
> to find all the matching files.
It's called iterator() and returns an Iterator<Resource> (in Java5
terms, but Ant is still at Java 1.3 or 1.4 so it is a plain Iterator).
See the org.apache.tools.ant.types.ResourceCollection interface that is
implemented by Union, Restrict and all the others.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Raja Nagendra Kumar <Na...@tejasoft.com>.
Hi Stefan,
Thank you very much for this info. I am trying to use this in our custom
task some thing like this
Union lUnion = (Union) TaskUtils.getCurrentProject().getReference(
lRoot);
Restrict lRestrict = new Restrict();
lRestrict.setProject(TaskUtils.getCurrentProject());
lRestrict.add(lUnion);
String[] patterns = normalisePattern(aFileName);
And and = new And();
for (String lPattern : patterns)
{
Name name = new Name();
name.setCaseSensitive(false);
name.setName(lPattern);
and.add(name);
lRestrict.add(and);
}
However, I am unable to find a way to get the values matching such an
expression. After doing the above, I am expecting some method in lRestrict
to find all the matching files. Some thing like DirectoryScanner.scan() and
then call to getIncludedFiles() gives the files which are matching the
criteria.
Could pl. let me know how should I get the matched files from the Restrict.
Regards,
Raja Nagendra Kumar
--
View this message in context: http://www.nabble.com/DirectoryScanner-cache-to-boost-performance-tp25422247p25454556.html
Sent from the Ant - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: DirectoryScanner cache to boost performance
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-09-13, Raja Nagendra Kumar <Na...@tejasoft.com> wrote:
> In this context, is there a way to let the DirectoryScanner read the entire
> file list only once in the entire ant session and every instance of
> DirectoryScanner and its scans() would use such set to find the files
> based on include and exclude patterns.
As long as all tasks you use can work on ResourceCollections, this may
work.
First, start out with a <fileset> for your directory and no excludes at
all.
fileset caches the DirectoryScanner instance and DirectoryScanner caches
its result, so in theory this would be a single scan then cache setup -
unfortunately it isn't. Most tasks grab into the fileset, pull out the
directory scanner and invoke scan on it, which clears the cached
results.
Note this has both historical and practical reasons:
* Directoryscanner predates FileSet by more than a year (and a point
release, FileSet was introduced in Ant 1.2) and both predate resource
collections by several years and releases.
* sometimes you really want a fileset to pick up changes you've made
during the build.
OK, back to the intial problem. The fileset alone won't help as I
pointed out, but there are resource collections that cache their
contents. One of them, is union. so
<union id="all-files">
<fileset dir="your-directory"/>
</union>
is a resource collection that holds all files below "your-directory" and
it will only ever scan the directory tree once (caching the results).
If you need to apply include/exclude patterns, use a restrict resource
collection and appropriate selectors like
<restrict id="java-files">
<resource refid="all-files"/>
<name name="**/*.java"/>
</restrict>
<restrict id="non-test-java-files">
<resource refid="all-files"/>
<and>
<name name="**/*.java"/>
<not>
<name name="**/*_Test*"/>
</not>
</and>
</restrict>
I think you get the idea.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org