You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/04/13 17:23:34 UTC
[GitHub] [lucene] rmuir opened a new pull request #82: LUCENE-9928: speed up analysis/icu regeneration
rmuir opened a new pull request #82:
URL: https://github.com/apache/lucene/pull/82
The compilation of the library is slow, disable optimization as it doesn't speed up our usage of the gennorm2 tool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir commented on pull request #82:
URL: https://github.com/apache/lucene/pull/82#issuecomment-820357230
I'm forcing --rerun-tasks when i do this. For the icu regeneration the primary input is the jar file version as I mentioned before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir merged pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir merged pull request #82:
URL: https://github.com/apache/lucene/pull/82
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir commented on pull request #82:
URL: https://github.com/apache/lucene/pull/82#issuecomment-820296271
@dweiss I'm using `regenerate -x bigDFA` quite often, and this icuCompileLinux task is my bottleneck. I've been running this quite often recently! This patch drops it from 160s -> 126s (see JIRA) on my main machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #82:
URL: https://github.com/apache/lucene/pull/82#issuecomment-820174362
Nice! Still - these should only run *very* infrequently, right? On 8x we regenerated every few... months? :) I'm sorry for delays on other issues, full queue of priorities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir commented on pull request #82:
URL: https://github.com/apache/lucene/pull/82#issuecomment-819533286
@dweiss, thanks for the assistance. using the additional cores shaved another 10s off on my mac.
bench of latest patch on mac (uses LLVM/clang vs GCC, has 6 cores):
```
before: 82.68 sec. compileIcuLinux
after: 53.11 sec. compileIcuLinux
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on a change in pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir commented on a change in pull request #82:
URL: https://github.com/apache/lucene/pull/82#discussion_r612641061
##########
File path: gradle/generation/icu.gradle
##########
@@ -215,6 +217,7 @@ configure(project(":lucene:analysis:icu")) {
project.quietExec {
executable "make"
workingDir icuSrcDir
+ // TODO: use tests.jvms or similar instead of hardcoding at 4
Review comment:
This is probably an easy win for folks with beefier machines, just outside of my gradle kung fu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a change in pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #82:
URL: https://github.com/apache/lucene/pull/82#discussion_r612692200
##########
File path: gradle/generation/icu.gradle
##########
@@ -215,6 +217,7 @@ configure(project(":lucene:analysis:icu")) {
project.quietExec {
executable "make"
workingDir icuSrcDir
+ // TODO: use tests.jvms or similar instead of hardcoding at 4
Review comment:
"-j${propertyOrDefault('tests.jvms', '4')" should work? You could also set it to processor count (via runtime)? tests.jvms may be a reasonable default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on a change in pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
rmuir commented on a change in pull request #82:
URL: https://github.com/apache/lucene/pull/82#discussion_r612869465
##########
File path: gradle/generation/icu.gradle
##########
@@ -215,6 +217,7 @@ configure(project(":lucene:analysis:icu")) {
project.quietExec {
executable "make"
workingDir icuSrcDir
+ // TODO: use tests.jvms or similar instead of hardcoding at 4
Review comment:
thanks i'll try the propertyOrDefault. I'd prefer not to use jvm's ActiveProcessorCount as i set it to 1 on purpose (because the build uses N jvms, each one should only try to use the resource of one core, so its not N^2). This way each of my N jvms only uses minimal compiler and gc threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #82: LUCENE-9928: speed up analysis/icu regeneration
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #82:
URL: https://github.com/apache/lucene/pull/82#issuecomment-820315980
Sure, no problem. If you reuse it often then you could also try to make it fully incremental by adding inputs and outputs to that task, then it'd be skipped if there's no need to recompile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org