You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Dawid Weiss <da...@gmail.com> on 2012/02/10 18:19:57 UTC
check-legal-lucene invoked repeatedly?
I made a simple test in which I did a noop for tests and ran ant test.
The build took 1 minute (trunk) and I could see slowdowns on:
check-legal-lucene
This seems to be costly and invoked multiple times with identical
arguments. I can look into this, but didn't want to miss anything --
is this how it's supposed to be?
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: check-legal-lucene invoked repeatedly?
Posted by Steven A Rowe <sa...@syr.edu>.
On 2/10/2012 at 1:27 PM, Uwe Schindler wrote:
> I would try to make this an ANT-taks by implementing ANTs interfaces, so
> it runs in the ANT JVM and does not spawn extra runtimes. For this check
> it's in my opinion easy to imeplement the task interface and iterate the
> fileset given.
+1
Re: check-legal-lucene invoked repeatedly?
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> I would try to make this an ANT-taks by implementing ANTs interfaces, so it
> runs in the ANT JVM and does not spawn extra runtimes. For this check it's
I have lots of experience in this area if you need anything, Uwe. You can also
peek and copy/paste from this task:
https://github.com/carrotsearch/randomizedtesting/blob/master/integration-ant/ant-junit4/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java
This has pretty much everything that's needed (complex subtypes,
resource collections, scanning over them etc.).
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: check-legal-lucene invoked repeatedly?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I would try to make this an ANT-taks by implementing ANTs interfaces, so it
runs in the ANT JVM and does not spawn extra runtimes. For this check it's
in my opinion easy to imeplement the task interface and iterate the fileset
given.
I can look into that. There are also other tasks running all the time in
separate JVMs that don't need to.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Friday, February 10, 2012 6:40 PM
> To: dev@lucene.apache.org
> Subject: Re: check-legal-lucene invoked repeatedly?
>
> The other issue I've had with check-legal is that it hangs (forever) any
build I try
> to run in the background on my linux box. No clue why, and no time to
look
> into it.
>
> ant test > t.out 2>&1 &
>
> I've just switched to running in the foreground and using tee to capture
the
> output.
>
> -Yonik
> lucidimagination.com
>
>
>
> On Fri, Feb 10, 2012 at 12:34 PM, Dawid Weiss
> <da...@cs.put.poznan.pl> wrote:
> >> No, it doesn't need to be invoked multiple times - it just needs to run
once
> per build session.
> >
> > That's what I thought. It gets invoked a number of times -- check this
out:
> > https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/1712/
> > consoleFull
> >
> > and these calls are relatively costly.
> >
> >> Maybe it could be hooked into lucene core's build, with a property that
gets
> set and checked so it runs just once?
> >
> > I think this makes sense. Do you know how to fix it, Steven? If you do
> > it may be faster than me grepping through the build files :)
> >
> > Dawid
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: check-legal-lucene invoked repeatedly?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
The other issue I've had with check-legal is that it hangs (forever)
any build I try to run in the background on my linux box. No clue
why, and no time to look into it.
ant test > t.out 2>&1 &
I've just switched to running in the foreground and using tee to
capture the output.
-Yonik
lucidimagination.com
On Fri, Feb 10, 2012 at 12:34 PM, Dawid Weiss
<da...@cs.put.poznan.pl> wrote:
>> No, it doesn't need to be invoked multiple times - it just needs to run once per build session.
>
> That's what I thought. It gets invoked a number of times -- check this out:
> https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/1712/consoleFull
>
> and these calls are relatively costly.
>
>> Maybe it could be hooked into lucene core's build, with a property that gets set and checked so it runs just once?
>
> I think this makes sense. Do you know how to fix it, Steven? If you do
> it may be faster than me grepping through the build files :)
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: check-legal-lucene invoked repeatedly?
Posted by Steven A Rowe <sa...@syr.edu>.
One 2/10/2012 at 1:29 PM, Steven A Rowe wrote:
> My current thinking is that the license checking code should be invoked
> once per module, and that each module can have a set of lib/ directories
> associated with it. By default, that set would consist only of the lib/
> directory associated with the module, but solr/core/ and lucene/core/
> would specialize the set: lucene/core/ would include lucene/lib/, and
> solr/core/ would include solr/lib/ and also all the example/**/lib/ dirs.
I put a patch up implementing these ideas at:
https://issues.apache.org/jira/browse/LUCENE-3774
Steve
RE: check-legal-lucene invoked repeatedly?
Posted by Steven A Rowe <sa...@syr.edu>.
Ideally the license checking java code would be a checked-in jar containing an ant task, so that it wouldn't need to be compiled on every build. I don't want to tackle that right now though.
In https://issues.apache.org/jira/browse/LUCENE-3267 Chris Male removed checking of contrib/queries/lib/ from check-legal-lucene's list of lib dirs, which apparently was otherwise not maintained at all, so that now only lucene/lib/ is ever checked. There two other existing lucene lib/ dirs are under the sandbox contrib and the demo contrib.
Solr, by contrast, has a more up-to-date list of directories to check, but it too suffers from module drift: contrib/langid/lib/ is missing, and dataimporthandler-extras/lib/ doesn't exist (anymore).
modules/ on trunk has no license checking coverage at all.
Per-module invocation, which I was considering, is problematic for Solr, since the example, which isn't exactly a module, has three separate lib/ directories.
My current thinking is that the license checking code should be invoked once per module, and that each module can have a set of lib/ directories associated with it. By default, that set would consist only of the lib/ directory associated with the module, but solr/core/ and lucene/core/ would specialize the set: lucene/core/ would include lucene/lib/, and solr/core/ would include solr/lib/ and also all the example/**/lib/ dirs.
Steve
> -----Original Message-----
> From: Steven A Rowe [mailto:sarowe@syr.edu]
> Sent: Friday, February 10, 2012 12:43 PM
> To: dev@lucene.apache.org
> Subject: RE: check-legal-lucene invoked repeatedly?
>
> I'll take a look. Actually, I remember thinking a while back that the
> best way to do this is to check each module's lib/ dir when the module is
> being tested, unlike the current setup, where all dirs are checked at
> once. That way people don't have to remember to update the global list.
>
> > -----Original Message-----
> > From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of
> > Dawid Weiss
> > Sent: Friday, February 10, 2012 12:35 PM
> > To: dev@lucene.apache.org
> > Subject: Re: check-legal-lucene invoked repeatedly?
> >
> > > No, it doesn't need to be invoked multiple times - it just needs to
> run
> > once per build session.
> >
> > That's what I thought. It gets invoked a number of times -- check this
> > out:
> > https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-
> > java7/1712/consoleFull
> >
> > and these calls are relatively costly.
> >
> > > Maybe it could be hooked into lucene core's build, with a property
> that
> > gets set and checked so it runs just once?
> >
> > I think this makes sense. Do you know how to fix it, Steven? If you do
> > it may be faster than me grepping through the build files :)
> >
> > Dawid
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
RE: check-legal-lucene invoked repeatedly?
Posted by Steven A Rowe <sa...@syr.edu>.
I'll take a look. Actually, I remember thinking a while back that the best way to do this is to check each module's lib/ dir when the module is being tested, unlike the current setup, where all dirs are checked at once. That way people don't have to remember to update the global list.
> -----Original Message-----
> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of
> Dawid Weiss
> Sent: Friday, February 10, 2012 12:35 PM
> To: dev@lucene.apache.org
> Subject: Re: check-legal-lucene invoked repeatedly?
>
> > No, it doesn't need to be invoked multiple times - it just needs to run
> once per build session.
>
> That's what I thought. It gets invoked a number of times -- check this
> out:
> https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-
> java7/1712/consoleFull
>
> and these calls are relatively costly.
>
> > Maybe it could be hooked into lucene core's build, with a property that
> gets set and checked so it runs just once?
>
> I think this makes sense. Do you know how to fix it, Steven? If you do
> it may be faster than me grepping through the build files :)
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
Re: check-legal-lucene invoked repeatedly?
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> No, it doesn't need to be invoked multiple times - it just needs to run once per build session.
That's what I thought. It gets invoked a number of times -- check this out:
https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/1712/consoleFull
and these calls are relatively costly.
> Maybe it could be hooked into lucene core's build, with a property that gets set and checked so it runs just once?
I think this makes sense. Do you know how to fix it, Steven? If you do
it may be faster than me grepping through the build files :)
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: check-legal-lucene invoked repeatedly?
Posted by Steven A Rowe <sa...@syr.edu>.
No, it doesn't need to be invoked multiple times - it just needs to run once per build session.
Maybe it could be hooked into lucene core's build, with a property that gets set and checked so it runs just once?
Steve
> -----Original Message-----
> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Sent: Friday, February 10, 2012 12:20 PM
> To: dev@lucene.apache.org
> Subject: check-legal-lucene invoked repeatedly?
>
> I made a simple test in which I did a noop for tests and ran ant test.
> The build took 1 minute (trunk) and I could see slowdowns on:
>
> check-legal-lucene
>
> This seems to be costly and invoked multiple times with identical
> arguments. I can look into this, but didn't want to miss anything --
> is this how it's supposed to be?
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org