You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steven Rowe (Created) (JIRA)" <ji...@apache.org> on 2012/02/05 22:17:59 UTC
[jira] [Created] (LUCENE-3753) Restructure the Lucene build system
Restructure the Lucene build system
-----------------------------------
Key: LUCENE-3753
URL: https://issues.apache.org/jira/browse/LUCENE-3753
Project: Lucene - Java
Issue Type: Improvement
Components: general/build
Affects Versions: 3.6, 4.0
Reporter: Steven Rowe
Assignee: Steven Rowe
Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201356#comment-13201356 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
bq. Can't say I object to the patch if it has a script to fix old patches
Yes.
bq. will go back to 3x too
That's my plan, yes.
bq. and if it will speed up the build system? (Note: I didn't dig into LUCENE-3754, but its my impression that it would?)
I don't know. I figured it would be about the same speed, but I'll check to be sure. (I think {{ant build-contrib}} in {{lucene/}} is the right thing to use for this.) LUCENE-3754 will speed up the Solr build, since it depends on lucene-core jar, but a lot (all?) of the Lucene build depends on un-jarred build outputs, and so won't be affected, AFAIK.
bq. can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.
I'll add them if they don't already work (not sure).
bq. can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up' i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool,
and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with the "real" conflicts then)
Yes, I agree a heads-up to the dev list first would be good - I was planning on giving 24 hours notice before committing.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3753) Restructure the Lucene
build system
Posted by "Uwe Schindler (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206065#comment-13206065 ]
Uwe Schindler edited comment on LUCENE-3753 at 2/11/12 9:27 AM:
----------------------------------------------------------------
Steven: There is still a problem with the clover reports: The directories for generating the reports don't correctly identify "Tests", "Test Results" and the "Class files". The ant task's "generate-clover-reports" seems to use the wrong paths. It looks like it only catches some tests, most test classes are listed under
We need maybe a more "universal" fileset for <testsources/> and <testresults/>. Maybe use the first one for contrib somehow globally:
{code:xml}
<fileset dir="contrib" id="clover.contrib.test.src.files">
<include name="**/test/**/*.java"/>
</fileset>
{code}
(e.g. using dir="."). Same for test-results, this is also wrong:
{code:xml}
<fileset dir="${build.dir}" id="clover.test.result.files">
<include name="**/test/TEST-*.xml" />
<!-- do not include BW tests -->
<exclude name="backwards/**"/>
</fileset>
{code}
Affects both 3.x and trunk. I don't want to fix it, you know better *where* al those different build dirs could live.
Ah, here are the reports: https://builds.apache.org/job/Lucene-trunk/clover/ (you can see the bugs when you look on the left frame and switch between the tabs "Classes", "Tests", "Results". Most tests are now listed at "Classes", which is wrong.
was (Author: thetaphi):
Steven: There is still a problem with the clover reports: The directories for generating the reports don't correctly identify "Tests", "Test Results" and the "Class files". The ant task's "generate-clover-reports" seems to use the wrong paths. It looks like it only catches some tests, most test classes are listed under
We need maybe a more "universal" fileset for <testsources/> and <testresults/>. Maybe use the first one for contrib somehow globally:
{code:xml}
<fileset dir="contrib" id="clover.contrib.test.src.files">
<include name="**/test/**/*.java"/>
</fileset>
{code}
(e.g. using dir="."). Same for test-results, this is also wrong:
{code:xml}
<fileset dir="${build.dir}" id="clover.test.result.files">
<include name="**/test/TEST-*.xml" />
<!-- do not include BW tests -->
<exclude name="backwards/**"/>
</fileset>
{code}
Affects both 3.x and trunk. I don't want to fix it, you know better *where* al those different build dirs could live.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201629#comment-13201629 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
bq. Interestingly I did a small surefire replacement for that ant task (with a few incompatibilities) that does run multiple jvms for testing. If you want to check it out and provide feedback, it'd be awesome.
Sure, I'll take a look later this week.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3753) Restructure the Lucene
build system
Posted by "Steven Rowe (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201530#comment-13201530 ]
Steven Rowe edited comment on LUCENE-3753 at 2/6/12 8:08 PM:
-------------------------------------------------------------
Final version of the patch:
* speeds up {{ant test}} from {{lucene/}} by making {{test-contrib}} depend on {{compile-test}} instead of {{build-contrib}} (jars don't need to be built for testing.)
* includes more {{\*.compiled}} property handling to avoid recompiling lucene-core and lucene-test-framework.
Post-patch {{ant compile-test}} times under {{lucene/}} are now *31s* after running {{ant clean}}, and *7s* when already compiled.
{{modules/}} and {{solr/}} timings did not change.
was (Author: steve_rowe):
Final version of the patch:
* speeds up {{ant compile-test}} from {{lucene/}} by making {{test-contrib}} depend on {{compile-test}} instead of {{build-contrib}} (jars don't need to be built for testing.)
* includes more {{\*.compiled}} property handling to avoid recompiling lucene-core and lucene-test-framework.
Post-patch {{ant compile-test}} times under {{lucene/}} are now *31s* after running {{ant clean}}, and *7s* when already compiled.
{{modules/}} and {{solr/}} timings did not change.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3753) Restructure the Lucene build system
Posted by "Steven Rowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3753:
--------------------------------
Attachment: LUCENE-3753.patch
This version of the patch adds:
* alias targets {{test-core}} and {{javadocs-core}} to {{lucene/core/build.xml}};
* alias target {{javadocs-core}} to {{lucene/test-framework/build.xml}}; and
* a new target {{compile-test}} to {{lucene/build.xml}}, which compiles all modules' non-test and test sources (the existing target {{build-contrib}} also builds contribs' jars).
Also, I added {{mkdir ${build.dir}}{{\}}} to the {{javadocs}} targets in {{lucene/core/}} and {{lucene/test-framework/}}, to make sure the target dir for the javadocs jars exists. (This isn't necessary in the current setup because the target dir is {{lucene/build/}}, which gets created when {{${javadoc.dir}}{{\}}} is created by these targets.)
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206451#comment-13206451 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
I committed clover test source pattern changes to trunk and branch_3x, but I haven't been able to access Jenkins since (service unavailable/proxy errors), so I'm not sure if they were effective. I'll keep checking.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3753) Restructure the Lucene
build system
Posted by "Steven Rowe (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201417#comment-13201417 ]
Steven Rowe edited comment on LUCENE-3753 at 2/6/12 5:35 PM:
-------------------------------------------------------------
I ran {{ant compile-test}} from {{modules/}} and {{solr/}}, both with and without running {{ant clean}} from the top level first (the timings below exclude {{ant clean}}) - these timings are best of 5 runs each:
||from {{modules/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|80s|80s|
|No|33s|33s|
||from {{solr/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|155s|155s|
|No|55s|54s|
These are basically unchanged.
was (Author: steve_rowe):
I ran {{ant compile-test}} from {{modules/}} and {{solr/}}, both with and without running {{ant clean}} from the top level first (the timings below exclude {{ant clean}}) - these timings are best of 5 runs each:
||from {{modules/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|80s|80s|
|No|33s|33s|
{panel}
||from {{solr/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|155s|155s|
|No|55s|54s|
These are basically unchanged.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201558#comment-13201558 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
Hi Dawid,
bq. Steve, are you going to merge running the tests somehow too?
No, my patch doesn't do that. Sounds really cool though. I think Mike M.'s python test runner does this?
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201553#comment-13201553 ]
Dawid Weiss commented on LUCENE-3753:
-------------------------------------
Steve, are you going to merge running the tests somehow too? I ask because I have a backlog of trying to integrate automatically load-balancing parallel tests; if we had a single test task this would mean one could pattern-scope the tests to be executed and (if not restricted) all of the tests would run in a single balancing scope (Mike pointed out that currently a lot of time is spent on waiting for the last test in each spawned).
Just asking, no obligations to change anything.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201371#comment-13201371 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
I ran {{ant build-contrib}} from {{lucene/}}, both with and without running {{ant clean}} first (the timings below exclude {{ant clean}}) - these timings are best of 5 runs each:
||{{ant clean}}?||Pre-patch||Post-patch||
|Yes|49s|44s|
|No|18s|19s|
So, slightly faster for the from-scratch case, and slightly slower from the already-built case - I think this slight slowdown is okay.
I'll run similar tests from {{modules/}} and {{solr/}} and report back.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206153#comment-13206153 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
Hi Uwe, I'm looking into it.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201580#comment-13201580 ]
Robert Muir commented on LUCENE-3753:
-------------------------------------
{quote}
The speedups aren't dramatic, especially after Robert worked on the longest tests, so this is not a huge priority.
{quote}
Well, from time to time we try to clean up the tests... but i think it only takes a few weeks for us to 'lose' all the benefits.
I'm not trying to say our tests get crappier, a lot of the reasons for the slowdown is really just more test coverage...!
Basically, I don't think we can view speeding up slow tests as one-off tasks... its gotta be a continuous maintenance thing.
For this reason, its important to speed up the other parts of the build (like rebuilding jars over and over again
such as this issue), and also to integrate the nice random test framework work etc you did... these are more 'permanent'
wins.
However, I'm just concerned about adopting mike's cheating, because I like the idea of 'ant test' being a total pain
to catch broken stuff. New modules, etc are added often, the build system is refactored, dependencies upgraded, etc etc,
and I'm concerned we would somehow screw ourselves up and release stuff with bogus dependencies.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201668#comment-13201668 ]
Uwe Schindler commented on LUCENE-3753:
---------------------------------------
+1 to commit this change to the build system! That was really needed!
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201546#comment-13201546 ]
Robert Muir commented on LUCENE-3753:
-------------------------------------
{quote}
Post-patch ant compile-test times under lucene/ are now 31s after running ant clean, and 7s when already compiled.
{quote}
Great! Thanks for doing this refactoring!
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201417#comment-13201417 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
I ran {{ant compile-test}} from {{modules/}} and {{solr/}}, both with and without running {{ant clean}} from the top level first (the timings below exclude {{ant clean}}) - these timings are best of 5 runs each:
||from {{modules/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|80s|80s|
|No|33s|33s|
{panel}
||from {{solr/}}||
||ant clean?||Pre-patch||Post-patch||
|Yes|155s|155s|
|No|55s|54s|
These are basically unchanged.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3753) Restructure the Lucene build system
Posted by "Steven Rowe (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe resolved LUCENE-3753.
---------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.6
The Jenkins builds look stable now.
Any additional problems can be addressed in new issues.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201613#comment-13201613 ]
Dawid Weiss commented on LUCENE-3753:
-------------------------------------
Nah, just read the docs -- you're right, this is not multiple-test-per-jvm allocation. I thought it was. Interestingly I did a small surefire replacement for that ant task (with a few incompatibilities) that does run multiple jvms for testing. If you want to check it out and provide feedback, it'd be awesome. An example of use is in this POM:
https://github.com/carrotsearch/randomizedtesting/blob/master/integration-maven/junit4-maven-plugin-tests/src/it/01-basic-test/pom.xml
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206065#comment-13206065 ]
Uwe Schindler commented on LUCENE-3753:
---------------------------------------
Steven: There is still a problem with the clover reports: The directories for generating the reports don't correctly identify "Tests", "Test Results" and the "Class files". The ant task's "generate-clover-reports" seems to use the wrong paths. It looks like it only catches some tests, most test classes are listed under
We need maybe a more "universal" fileset for <testsources/> and <testresults/>. Maybe use the first one for contrib somehow globally:
{code:xml}
<fileset dir="contrib" id="clover.contrib.test.src.files">
<include name="**/test/**/*.java"/>
</fileset>
{code}
(e.g. using dir="."). Same for test-results, this is also wrong:
{code:xml}
<fileset dir="${build.dir}" id="clover.test.result.files">
<include name="**/test/TEST-*.xml" />
<!-- do not include BW tests -->
<exclude name="backwards/**"/>
</fileset>
{code}
Affects both 3.x and trunk. I don't want to fix it, you know better *where* al those different build dirs could live.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203123#comment-13203123 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
Committed to branch_3x.
Now working on fixing broken target {{generate-maven-artifacts}} under {{lucene/}} on trunk.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201600#comment-13201600 ]
Dawid Weiss commented on LUCENE-3753:
-------------------------------------
@Robert. You're right, dependencies (and their order) may screw up things. I was tinkering with an idea that testing could be a multi-pass process, then it would be possible to collect all the "specs" in one pass and then run everything in another pass... but it temporarily exceeds my time limits to actually implement this :)
@Steven. That was my impression from reading the release notes and from looking at the code:
{quote}
Surefire 2.12 now supports parallel forks, which can also be run
in runOrder="balanced", which is a nice feature for long-running
tests.
{quote}
Ah, I see -- I didn't mean "balanced" in the sense of dynamic balancing, the tests are statically assigned, but they do run in separate concurrent slave JVMs and that's still something
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204024#comment-13204024 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
{quote}
bq. Now working on fixing broken target generate-maven-artifacts under lucene/ on trunk.
This is now fixed - I committed the fix to branch_3x as well, where for some reason generate-maven-artifacts was already functional without the fix....
{quote}
I understand now why {{generate-maven-artifacts}} was functional in branch_3x *before I committed the backport*: as a result of {{svn move}}'ing files, the original directories are retained until {{svn commit}} is done.
I've seen this enough times now that I should remember: committing to subversion can change build behavior.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3753) Restructure the Lucene build system
Posted by "Steven Rowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3753:
--------------------------------
Attachment: LUCENE-3753.patch
Patch implementing the idea, along with a script to fix existing patches against the old structure to be against the new structure.
Run this svn move script before applying the patch:
{noformat}
svn mv --parents lucene/src/java lucene/core/src/java
svn mv --parents lucene/src/test lucene/core/src/test
svn mv --parents lucene/src/resources lucene/core/src/resources
svn mv lucene/src/site lucene/site
svn mv --parents lucene/src/test-framework/java lucene/test-framework/src/java
svn mv --parents lucene/src/test-framework/resources lucene/test-framework/src/resources
svn mv --parents lucene/src/tools/java lucene/tools/src/java
svn mv --parents lucene/src/tools/javadoc lucene/tools/javadoc
svn mv --parents lucene/src/tools/prettify lucene/tools/prettify
svn rm lucene/src
svn mv --parents dev-tools/maven/lucene/src/pom.xml.template dev-tools/maven/lucene/core/pom.xml.template
svn mv --parents dev-tools/maven/lucene/src/test-framework/pom.xml.template dev-tools/maven/lucene/test-framework/pom.xml.template
svn rm dev-tools/maven/lucene/src
{noformat}
I think this is ready to go.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201559#comment-13201559 ]
Robert Muir commented on LUCENE-3753:
-------------------------------------
{quote}
if we had a single test task this would mean one could pattern-scope the tests to be executed and (if not restricted) all of the tests would run in a single balancing scope (Mike pointed out that currently a lot of time is spent on waiting for the last test in each spawned).
{quote}
Yes, but this doesn't really test any dependencies that different contribs/modules might have right? Or maybe i misunderstand what you are suggesting.
I think Mike's python runner "cheats" and makes a huge classpath... sure its faster, but, it loses test coverage because modules could have
bogus dependencies and we would never know.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201575#comment-13201575 ]
Dawid Weiss commented on LUCENE-3753:
-------------------------------------
Yes, this is what Mike's python magic does -- puts a fraction of all tests on each slave, then assigns on-demand. I reimplemented this in Java and it works quite nice as it can even consume precomputed statistics to make better estimates. The problem is as always in details (which I won't trouble you with at the moment). Robert is also right that re-running with different parameters/ JVM settings/ classpath will require multiple runs... for a moment I had a hope to neglect this :)
The speedups aren't dramatic, especially after Robert worked on the longest tests, so this is not a huge priority. Interestingly, Maven's latest release of surefire also includes an option to do balanced parallel tests (on separate vms). Didn't experiment with it yet.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201579#comment-13201579 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
bq. Maven's latest release of surefire also includes an option to do balanced parallel tests (on separate vms).
I'm not sure that's exactly true? [SUREFIRE-799|http://jira.codehaus.org/browse/SUREFIRE-799?focusedCommentId=289606&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-289606] includes a discussion of some limitations of the new capabilities.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201317#comment-13201317 ]
Robert Muir commented on LUCENE-3753:
-------------------------------------
Can't say I object to the patch if it has a script to fix old patches, will go back to 3x too, and if it will speed up the build system?
(Note: I didn't dig into LUCENE-3754, but its my impression that it would?)
My only suggestions (more like personal requests actually):
* can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work
already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.
* can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up'
i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool,
and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with
the "real" conflicts then)
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201604#comment-13201604 ]
Dawid Weiss commented on LUCENE-3753:
-------------------------------------
I meant to end with a "?" -- do you know if my understanding is correct, Steve?
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3753) Restructure the Lucene build system
Posted by "Steven Rowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3753:
--------------------------------
Attachment: LUCENE-3753.patch
Final version of the patch:
* speeds up {{ant compile-test}} from {{lucene/}} by making {{test-contrib}} depend on {{compile-test}} instead of {{build-contrib}} (jars don't need to be built for testing.)
* includes more {{\*.compiled}} property handling to avoid recompiling lucene-core and lucene-test-framework.
Post-patch {{ant compile-test}} times under {{lucene/}} are now *31s* after running {{ant clean}}, and *7s* when already compiled.
{{modules/}} and {{solr/}} timings did not change.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3753) Restructure the Lucene
build system
Posted by "Steven Rowe (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201356#comment-13201356 ]
Steven Rowe edited comment on LUCENE-3753 at 2/6/12 3:59 PM:
-------------------------------------------------------------
bq. Can't say I object to the patch if it has a script to fix old patches
Yes.
bq. will go back to 3x too
That's my plan, yes.
bq. and if it will speed up the build system? (Note: I didn't dig into LUCENE-3754, but its my impression that it would?)
I don't know. I figured it would be about the same speed, but I'll check to be sure. (I think {{ant build-contrib}} in {{lucene/}} is the right thing to use for this.) LUCENE-3754 will speed up the Solr build, since it depends on lucene-core jar, but a lot (all?) of the Lucene build depends on un-jarred build outputs, and so won't be affected, AFAIK.
bq. can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.
I'll add them if they don't already work (not sure).
bq. can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up' i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool, and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with the "real" conflicts then)
Yes, I agree a heads-up to the dev list first would be good - I was planning on giving 24 hours notice before committing.
was (Author: steve_rowe):
bq. Can't say I object to the patch if it has a script to fix old patches
Yes.
bq. will go back to 3x too
That's my plan, yes.
bq. and if it will speed up the build system? (Note: I didn't dig into LUCENE-3754, but its my impression that it would?)
I don't know. I figured it would be about the same speed, but I'll check to be sure. (I think {{ant build-contrib}} in {{lucene/}} is the right thing to use for this.) LUCENE-3754 will speed up the Solr build, since it depends on lucene-core jar, but a lot (all?) of the Lucene build depends on un-jarred build outputs, and so won't be affected, AFAIK.
bq. can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.
I'll add them if they don't already work (not sure).
bq. can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up' i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool,
and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with the "real" conflicts then)
Yes, I agree a heads-up to the dev list first would be good - I was planning on giving 24 hours notice before committing.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203176#comment-13203176 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
bq. Now working on fixing broken target {{generate-maven-artifacts}} under {{lucene/}} on trunk.
This is now fixed - I committed the fix to branch_3x as well, where for some reason {{generate-maven-artifacts}} was already functional without the fix....
I'll leave this issue open for a day or two in case more problems crop up.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3753) Restructure the Lucene build
system
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202709#comment-13202709 ]
Steven Rowe commented on LUCENE-3753:
-------------------------------------
Committed to trunk.
Working on backport to branch_3x now.
> Restructure the Lucene build system
> -----------------------------------
>
> Key: LUCENE-3753
> URL: https://issues.apache.org/jira/browse/LUCENE-3753
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/build
> Affects Versions: 3.6, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Attachments: LUCENE-3753.patch, LUCENE-3753.patch, LUCENE-3753.patch
>
>
> Split out separate core/, test-framework/, and tools/ modules, each with its own build.xml, under the lucene/ directory, similar to the Solr restructuring done in SOLR-2452.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org