You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2025/10/30 18:32:21 UTC

[PR] Speed up java formatting tasks [lucene]

dweiss opened a new pull request, #15386:
URL: https://github.com/apache/lucene/pull/15386

   Google java format is quite costly. When a full build is triggered (for some reason), it takes significant resources to check and reformat files that couldn't possibly have changed. 
   
   I've added a local file state cache to both tasks that invoke google java format. Each file's state is based on its length and last modification timestamp (much like rsync does it). Any files for which the cache indicates no changes will be omitted in checks. 
   
   This makes google java format practically a zero-cost operation after the first initial run (unless you clean the build folders, that is).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3475912821

   Most of it is likely to be startup time. Formatting one file should be super-fast. I think you either need to play with jvm settings or make a simple http server...
   
   I myself don't care that much about formatting until I get the tests to pass, then just run ```gradlew tidy check -x test``` on the entire project to ensure everything (excluding tests) complies.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #15386:
URL: https://github.com/apache/lucene/pull/15386


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3475941418

   I also used IndexWriter.java in my example which is not a small file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3472465736

   Thank you, this totally works. For VSCode case, I think this solution is still too slow as it will exceed `editor.formatOnSaveTimeout` default of 750ms. But I can experiment with it on-demand for now in vim and see what the options are, including cranking that limit.
   
   ```
   $ time google-java-format --skip-reflowing-long-strings -i lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
   
   real	0m1.751s
   user	0m5.263s
   sys	0m0.093s
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3470462476

   slightly related, I can try to dig more into it later, but I've been unsuccessful at figuring out the correct incantation to `google-java-format` for our sources.
   
   purpose would be to make it possible to configure format-on-save workflow for vim, vscode, etc.
   
   The tool is not particularly fast, but it is fast enough when only formatting a single file... the one you just edited. 
   
   ```
   lucene[main]$ google-java-format -i lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
   lucene[main]$ git diff
   diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
   index aa2c92ffecc..026f339956f 100644
   --- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
   +++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
   @@ -637,7 +637,8 @@ public class IndexWriter
                          MergeTrigger.GET_READER,
                          sci -> {
                            assert stopCollectingMergedReaders.get() == false
   -                            : "illegal state  merge reader must be not pulled since we already stopped waiting for merges";
   +                            : "illegal state  merge reader must be not pulled since we already"
   +                                  + " stopped waiting for merges";
                            SegmentReader apply = readerFactory.apply(sci);
                            mergedReaders.put(sci.info.name, apply);
                            // we need to incRef the files of the opened SR otherwise it's possible that
   ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3471652619

   I think this is the relevant bit -
   
   https://github.com/apache/lucene/blob/main/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/spotless/ParentGoogleJavaFormatTask.java#L132-L136
   
   String breaking in google-java-format adds another pass and can mess things up - I have mixed experience with this.
   
   Adding ```--replace --skip-reflowing-long-strings``` should work. If I run this:
   
   ```
   $ find . -name "*.java" -print0 | xargs -0 -P 4 -n 500 java -jar ~/tmp/google-java-format-1.32.0-all-deps.jar --replace --skip-reflowing-long-strings
   $ git status
   ```
   
   I get just the Java files we generate (and ignore in our own builds).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Speed up java formatting tasks [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15386:
URL: https://github.com/apache/lucene/pull/15386#issuecomment-3476621433

   Something else you could try - google java format has native/binary releases compiled using graalvm:
   https://github.com/google/google-java-format/releases
   
   I see shorter execution times on linux compared to full jvm startup.
   ```
   $ time ./google-java-format_linux-x86-64 --skip-reflowing-long-strings -i lucene/lucene/core/src/java/org/apache/lucene/index/IndexWriter
   
   real	0m0.422s
   user	0m0.326s
   sys	0m0.096s
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org