You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2020/07/15 04:35:23 UTC
[GitHub] [lucenenet] Shazwazza opened a new pull request #310: WIP Benchmarks between nuget versions
Shazwazza opened a new pull request #310:
URL: https://github.com/apache/lucenenet/pull/310
This is a WIP for creating benchmarkdotnet tests to see performance (currently for testing between nuget packages).
I have asked about specifying different nuget package sources here https://github.com/dotnet/BenchmarkDotNet/pull/922#issuecomment-658519567 (I feel like I had done this in some way when implementing that feature but can't remember now) since that will be needed to test the 'current' version. It is not possible to reference the 'current' projects and then also use 'WithNuget' because that results in Nuget errors.
Currently just added a single benchmark as an example, this is just the script from the home page. There's manual iterations specified in there so that we don't get warnings from BenchmarkDotNet about `MultimodalDistribution` or `MinIterationTime` since one less iterations are too fast to measure correctly.
This example benchmark isn't showing much since the results are nearly identical between versions, for example:
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.400-preview-015151
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
IterationCount=15 LaunchCount=2 WarmupCount=10
```
| Method | Job | NuGetReferences | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|--------------- |---------------- |------------------------------------------- |---------:|--------:|--------:|-----------:|----------:|----------:|----------:|
| HomePageScript | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007 | 207.5 ms | 3.94 ms | 5.65 ms | 63000.0000 | 6000.0000 | 1000.0000 | 249.37 MB |
| HomePageScript | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008 | 196.8 ms | 6.20 ms | 9.28 ms | 60000.0000 | 9000.0000 | 2000.0000 | 250.74 MB |
| HomePageScript | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009 | 208.8 ms | 7.15 ms | 9.54 ms | 62000.0000 | 7000.0000 | 2000.0000 | 250.77 MB |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on pull request #310: WIP Benchmarks between nuget versions
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-661002167
Thanks for putting this together.
I suspect the reason you are not seeing an improvement between versions is because you are using a `RAMDirectory`, which bypasses the codecs and some other components that we have improved.
I added benchmarks using the `IndexFiles` and `SearchFiles` from `Lucene.Net.Demo` as templates. These are set up to use a fixed seed to generate the same data over and over again so we don't have to add several megabytes of test files to the repo just to do a benchmark.
Index speed doesn't seem to have improved, but search speed has doubled since 4.8.0-beta00005 (the one with the most downloads on NuGet.org). Here are the results for .NET Core 3.1 on on Windows.
## Indexing
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
```
| Method | Job | NuGetReferences | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|----------- |---------------- |------------------------------------------- |---------:|---------:|---------:|-----------:|----------:|----------:|----------:|
| IndexFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005 | 734.9 ms | 37.14 ms | 55.58 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09 MB |
| IndexFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006 | 703.7 ms | 13.92 ms | 20.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.07 MB |
| IndexFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007 | 705.3 ms | 15.95 ms | 23.38 ms | 43000.0000 | 8000.0000 | 7000.0000 | 221 MB |
| IndexFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008 | 747.0 ms | 42.43 ms | 59.47 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.39 MB |
| IndexFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009 | 725.3 ms | 25.59 ms | 36.70 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.31 MB |
| IndexFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010 | 724.1 ms | 15.41 ms | 22.10 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.23 MB |
## Searching
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
IterationCount=100 LaunchCount=3 WarmupCount=15
```
| Method | Job | NuGetReferences | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------ |---------------- |---------------------------------------------------------------------------------- |----------:|---------:|----------:|----------:|----------:|----------:|------:|----------:|
| SearchFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005,Lucene.Net.QueryParser 4.8.0-beta00005 | 171.18 ms | 2.004 ms | 10.283 ms | 169.51 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006,Lucene.Net.QueryParser 4.8.0-beta00006 | 169.86 ms | 2.666 ms | 13.754 ms | 168.14 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007,Lucene.Net.QueryParser 4.8.0-beta00007 | 173.67 ms | 3.417 ms | 17.595 ms | 169.20 ms | 9000.0000 | 1000.0000 | - | 41.26 MB |
| SearchFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008,Lucene.Net.QueryParser 4.8.0-beta00008 | 95.97 ms | 1.178 ms | 6.068 ms | 95.84 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009,Lucene.Net.QueryParser 4.8.0-beta00009 | 93.31 ms | 0.776 ms | 3.890 ms | 93.60 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010,Lucene.Net.QueryParser 4.8.0-beta00010 | 85.79 ms | 0.307 ms | 1.552 ms | 85.70 ms | 8666.6667 | 500.0000 | - | 40.19 MB |
It would be good to set up a stage on Azure DevOps to run these (off by default) so we can see the results on different operating systems and target frameworks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 commented on pull request #310: WIP Benchmarks between nuget versions
Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-823894606
I have rebased against the master branch so this is up to date again.
I also rearranged the project so all of the NuGet versions are defined in `BuildConfigurations.cs`. I also got it working (sort of) with a [local build](https://github.com/dotnet/BenchmarkDotNet/issues/1486) from source so it can be compared against the NuGet packages. Unfortunately, I couldn't find any documentation of what the defaults are for the properties of a `Release` build, so this appears to be basically building a `Debug` build with optimizations turned on. Requires more research to set all of the `Release` properties when the configuration is `LocalBuild`.
This isn't quite up to the state which we can merge it, though - we still need a top-level `benchmarks` folder as mentioned in #349 and to reorganize the project as well as rename this from `Lucene.Net.Tests.BenchmarkDotNet` to something that doesn't include the word `Tests`, but it is a few steps closer to where we need to be and it is quicker to update all of the benchmarks with a new set of NuGet packages now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] NightOwl888 edited a comment on pull request #310: WIP Benchmarks between nuget versions
Posted by GitBox <gi...@apache.org>.
NightOwl888 edited a comment on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-823894606
I have rebased against the master branch so this is up to date again.
I also rearranged the project so all of the NuGet versions are defined in `BuildConfigurations.cs`. I also got it working (sort of) with a [local build](https://github.com/dotnet/BenchmarkDotNet/issues/1486) from source so it can be compared against the NuGet packages. Unfortunately, I couldn't find any documentation of what the defaults are for the properties of a `Release` build, so this appears to be basically building a `Debug` build with optimizations turned on.
As you can see by the above image, the performance of the `LocalBuild` is much worse than using the NuGet packages. Requires more research to set all of the `Release` properties when the configuration is `LocalBuild`.
This isn't quite up to the state which we can merge it, though - we still need a top-level `benchmarks` folder as mentioned in #349 and to reorganize the project as well as rename this from `Lucene.Net.Tests.BenchmarkDotNet` to something that doesn't include the word `Tests`, but it is a few steps closer to where we need to be and it is quicker to update all of the benchmarks with a new set of NuGet packages now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org