You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2023/10/12 10:43:06 UTC
[PR] Initial take at adding JMH micro-benchmarks [lucene]
dweiss opened a new pull request, #12663:
URL: https://github.com/apache/lucene/pull/12663
As per discussion in: https://github.com/apache/lucene/issues/12641
This patch adds the ability to compile JMH microbenchmarks and run them from within Lucene codebase. I didn't use any plugins as they, in my opinion, obscure the view of what's happening and what's needed. I also wanted it to work with the module system and without uber-jars.
```
gradlew -p lucene/benchmark-jmh assemble
```
compiles microbenchmarks with JMH, then displays a short help on what to do next:
```
JMH benchmarks compiled. Run them with:
java -jar lucene\benchmark-jmh\build\benchmarks\lucene-benchmark-jmh-10.0.0-SNAPSHOT.jar
or
java --module-path lucene\benchmark-jmh\build\benchmarks --module org.apache.lucene.benchmark.jmh
JMH options you can use with the above:
-h displays verbose help for all options
-l list available benchmarks
-lp list benchmarks that pass the filter and their parameters
regexp execute all benchmark containing regexp
```
I didn't add any gradle-specific tasks to run the benchmarks directly - it's technically easy but it seems like a bad idea to fork a subprocess from (quite heavy) gradle when such delicate things are being measured? I'm not sure here.
Some future improvements/ remaining issues include:
1) multi-source compilation so that code can be compiled against future APIs (not the one gradle runs with).
2) forbiddenapis doesn't have access to vectors and I had to turn it off,
3) jmh is GPL. It's a dev tool, I don't think it's a problem. I left license checks out of it for the time being.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759607611
@dweiss this is awesome!!! Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759777296
@benwtrent Can you take a look? If there's any wording you think would work better, please do change it. Explaining how JMH works is perhaps beyond the scope of the readme file... ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759752950
@dweiss one suggestion I have is adding a `README` for `lucene/benchmark-jmh` on how to run the benchmarks and add new ones. Your description on how to use it in this PR seems adequate.
I tested this locally and was able to follow your directions and get the benchmarks to run without issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #12663:
URL: https://github.com/apache/lucene/pull/12663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759795868
Looks good @dweiss I added a line with an example for running a single benchmark or suite of benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759605550
Let me review the minor remaining bits - I had to leave the computer before the tests completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #12663:
URL: https://github.com/apache/lucene/pull/12663#discussion_r1356970803
##########
lucene/benchmark-jmh/README.txt:
##########
@@ -0,0 +1,21 @@
+The :lucene:benchmark-jmh module contains can be used to compile
+and execute JMH (https://github.com/openjdk/jmh) micro-benchmarks.
+
+Look at existing classes and JMH documentation for inspiration on how
+to write good micro-benchmarks.
+
+To compile the project and prepare JMH launcher, run:
+
+gradlew :lucene:benchmark-jmh:assemble
+
+The above target will display exact commands to execute JMH from
+command line, for example:
+
+java --module-path lucene\benchmark-jmh\build\benchmarks --module org.apache.lucene.benchmark.jmh
+
+You can pass any JMH options to the above command, for example:
+
+ -h displays verbose help for all options
+ -l list available benchmarks
+ -lp list benchmarks that pass the filter and their parameters
+ regexp execute all benchmark containing regexp
Review Comment:
```suggestion
regexp execute all benchmark containing regexp
Here is an example running a single benchmark:
java --module-path lucene\benchmark-jmh\build\benchmarks --module org.apache.lucene.benchmark.jmh org.apache.lucene.benchmark.jmh.BinaryCosineBenchmark.cosineDistanceNew
Or running all of BinaryCosineBenchmark
java --module-path lucene\benchmark-jmh\build\benchmarks --module org.apache.lucene.benchmark.jmh org.apache.lucene.benchmark.jmh.BinaryCosineBenchmark
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12663:
URL: https://github.com/apache/lucene/pull/12663#discussion_r1357235935
##########
lucene/benchmark-jmh/README.txt:
##########
@@ -0,0 +1,21 @@
+The :lucene:benchmark-jmh module contains can be used to compile
+and execute JMH (https://github.com/openjdk/jmh) micro-benchmarks.
+
+Look at existing classes and JMH documentation for inspiration on how
+to write good micro-benchmarks.
+
+To compile the project and prepare JMH launcher, run:
+
+gradlew :lucene:benchmark-jmh:assemble
+
+The above target will display exact commands to execute JMH from
+command line, for example:
+
+java --module-path lucene\benchmark-jmh\build\benchmarks --module org.apache.lucene.benchmark.jmh
+
+You can pass any JMH options to the above command, for example:
+
+ -h displays verbose help for all options
+ -l list available benchmarks
+ -lp list benchmarks that pass the filter and their parameters
+ regexp execute all benchmark containing regexp
Review Comment:
Oh, apologies - I thought you added that change to the branch, not as github's suggested change. I added this manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759758762
I enabled forbiddenapis to the extent possible, to make Uwe happier. ;) I like those flame-graphs that Solr has, etc., but these can come later. I'll add a simple ```gradlew :helpJmh``` target similar to everything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Initial take at adding JMH micro-benchmarks [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #12663:
URL: https://github.com/apache/lucene/pull/12663#issuecomment-1759558774
@dweiss thank you for this! I tried it out and it works great. Glad to see the `cosineDistanceVectorUtil()` working in this example which is exactly what I want to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org