You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2011/07/24 23:40:20 UTC
[jira] [Created] (LUCENE-3337) avoid building jar files unless
necessary in build
avoid building jar files unless necessary in build
--------------------------------------------------
Key: LUCENE-3337
URL: https://issues.apache.org/jira/browse/LUCENE-3337
Project: Lucene - Java
Issue Type: Bug
Components: general/build
Affects Versions: 3.4, 4.0
Reporter: Robert Muir
This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073199#comment-13073199 ]
Steven Rowe commented on LUCENE-3337:
-------------------------------------
On SOLR-2452, yonik wrote:
bq. On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51.
And then on this issue:
bq. Sweet, "ant example" the second time now comes in at 16sec for me!
Cool, twice as fast as it was pre-SOLR-2452!
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3337:
--------------------------------
Attachment: LUCENE-3337.patch
updated state, working on more of the solr deps to try to improve the situation... there is still a lot more to do here though, but it seems to improve a bit, at least we are only building jar files once (but in many cases like 'test' we never need to do this in the first place)
||task||trunk||patch||
|ant example (clean checkout)|30s|20s|
|ant example (already compiled)|18s|10s|
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3337:
--------------------------------
Attachment: LUCENE-3337.patch
{quote}
I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build.
{quote}
The properties were being passed in as intended, but the {{<uptodate>}} tasks were still running, since their execution was not dependent on the already-defined values of {{\*.uptodate}} output properties.
The attached version of the patch, which includes Robert's latest patch, wraps the {{<uptodate>}} tasks in {{check-\*\-uptodate}} targets, and makes the {{compile-\*}} targets depend on them, so that the {{<uptodate>}} tasks only run when the corresponding {{\*.update}} property value has not previously been set.
The only problem with this approach is that the {{\*.jar}} properties aren't constructed properly, since the property passing scheme only works one-way (properties are passed into {{<ant>}} invocations, but not back out of them). I worked around this problem by defining the {{\*.jar}} properties statically.
My build times now:
||task||trunk||patch||patch w/property passing optimizations||
|ant example (clean checkout)|180s|119s|93s|
|ant example (already compiled)|132s|61s|55s|
This shaves a few more seconds off my build time.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071568#comment-13071568 ]
Robert Muir commented on LUCENE-3337:
-------------------------------------
whoah, those timings look great!
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3337:
--------------------------------
Attachment: LUCENE-3337.patch
here's the start to a patch, i started with solr's core/
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072401#comment-13072401 ]
Yonik Seeley commented on LUCENE-3337:
--------------------------------------
Sweet, "ant example" the second time now comes in at 16sec for me!
This helps big time doing ad-hoc testing/debugging using the server.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe updated LUCENE-3337:
--------------------------------
Attachment: LUCENE-3337.patch
This version of the patch moves uptodate checks and targets to build jars for other modules to {{lucene/contrib/contrib-build.xml}}. All Solr & Lucene modules are now covered. The Lucene core build now has a {{jar-core}} specialization that avoids creating the jar when it's already up-to-date - previously when running {{ant test}} from {{solr/}}, the Lucene core jar was built 41 times, because the manifest was newer each time...
My build times now:
||task||trunk||Robert's patch||My patch w/property passing optimizations||
|ant example (clean checkout)|180s|119s|52s|
|ant example (already compiled)|132s|61s|27s|
I think this should be committed; further changes can be made with more patches.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071975#comment-13071975 ]
Robert Muir commented on LUCENE-3337:
-------------------------------------
{quote}
I think this should be committed; further changes can be made with more patches.
{quote}
+1 to commit!
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072185#comment-13072185 ]
Steven Rowe commented on LUCENE-3337:
-------------------------------------
Committed:
- r1151720: trunk
- r1151735: branch_3x
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071084#comment-13071084 ]
Steven Rowe commented on LUCENE-3337:
-------------------------------------
Robert, your patch is a collection of optimizations to the SOLR-2452 work. Looks good!
My build times:
||task||trunk||patch||
|ant example (clean checkout)|180s|119s|
|ant example (already compiled)|132s|61s|
So I'm seeing the same kind of speedups as you. +1 to commit - we could use this issue to host a series of further patches.
I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Rowe resolved LUCENE-3337.
---------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.4
Assignee: Steven Rowe
Lucene Fields: [New, Patch Available] (was: [New])
Closing this issue - further work can be done in new issues.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Assignee: Steven Rowe
> Fix For: 3.4, 4.0
>
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071169#comment-13071169 ]
Steven Rowe commented on LUCENE-3337:
-------------------------------------
When I look at the output from {{ant -v example}} after applying my version of the patch, I can see that the {{compile-test}} target is called 16 times from {{lucene/build.xml}}. Seems like it could be sped up through more/better property passing. And why should building a module's jar trigger compilation of Lucene's tests?
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3337) avoid building jar files unless
necessary in build
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3337:
--------------------------------
Attachment: LUCENE-3337.patch
some other minor optimizations, a few seconds here or there.
> avoid building jar files unless necessary in build
> --------------------------------------------------
>
> Key: LUCENE-3337
> URL: https://issues.apache.org/jira/browse/LUCENE-3337
> Project: Lucene - Java
> Issue Type: Bug
> Components: general/build
> Affects Versions: 3.4, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3337.patch, LUCENE-3337.patch, LUCENE-3337.patch
>
>
> This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org