You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2025/10/30 18:32:21 UTC
[PR] Speed up java formatting tasks [lucene]
dweiss opened a new pull request, #15386:
URL: https://github.com/apache/lucene/pull/15386
Google java format is quite costly. When a full build is triggered (for some reason), it takes significant resources to check and reformat files that couldn't possibly have changed.
I've added a local file state cache to both tasks that invoke google java format. Each file's state is based on its length and last modification timestamp (much like rsync does it). Any files for which the cache indicates no changes will be omitted in checks.
This makes google java format practically a zero-cost operation after the first initial run (unless you clean the build folders, that is).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3475912821
Most of it is likely to be startup time. Formatting one file should be super-fast. I think you either need to play with jvm settings or make a simple http server...
I myself don't care that much about formatting until I get the tests to pass, then just run ```gradlew tidy check -x test``` on the entire project to ensure everything (excluding tests) complies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #15386:
URL: https://github.com/apache/lucene/pull/15386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3475941418
I also used IndexWriter.java in my example which is not a small file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3472465736
Thank you, this totally works. For VSCode case, I think this solution is still too slow as it will exceed `editor.formatOnSaveTimeout` default of 750ms. But I can experiment with it on-demand for now in vim and see what the options are, including cranking that limit.
```
$ time google-java-format --skip-reflowing-long-strings -i lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
real 0m1.751s
user 0m5.263s
sys 0m0.093s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3470462476
slightly related, I can try to dig more into it later, but I've been unsuccessful at figuring out the correct incantation to `google-java-format` for our sources.
purpose would be to make it possible to configure format-on-save workflow for vim, vscode, etc.
The tool is not particularly fast, but it is fast enough when only formatting a single file... the one you just edited.
```
lucene[main]$ google-java-format -i lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
lucene[main]$ git diff
diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
index aa2c92ffecc..026f339956f 100644
--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -637,7 +637,8 @@ public class IndexWriter
MergeTrigger.GET_READER,
sci -> {
assert stopCollectingMergedReaders.get() == false
- : "illegal state merge reader must be not pulled since we already stopped waiting for merges";
+ : "illegal state merge reader must be not pulled since we already"
+ + " stopped waiting for merges";
SegmentReader apply = readerFactory.apply(sci);
mergedReaders.put(sci.info.name, apply);
// we need to incRef the files of the opened SR otherwise it's possible that
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3471652619
I think this is the relevant bit -
https://github.com/apache/lucene/blob/main/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/spotless/ParentGoogleJavaFormatTask.java#L132-L136
String breaking in google-java-format adds another pass and can mess things up - I have mixed experience with this.
Adding ```--replace --skip-reflowing-long-strings``` should work. If I run this:
```
$ find . -name "*.java" -print0 | xargs -0 -P 4 -n 500 java -jar ~/tmp/google-java-format-1.32.0-all-deps.jar --replace --skip-reflowing-long-strings
$ git status
```
I get just the Java files we generate (and ignore in our own builds).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Speed up java formatting tasks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3476621433
Something else you could try - google java format has native/binary releases compiled using graalvm:
https://github.com/google/google-java-format/releases
I see shorter execution times on linux compared to full jvm startup.
```
$ time ./google-java-format_linux-x86-64 --skip-reflowing-long-strings -i lucene/lucene/core/src/java/org/apache/lucene/index/IndexWriter
real 0m0.422s
user 0m0.326s
sys 0m0.096s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org