You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2020/07/20 12:35:02 UTC
[GitHub] [lucenenet] NightOwl888 commented on pull request #310: WIP Benchmarks between nuget versions
NightOwl888 commented on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-661002167
Thanks for putting this together.
I suspect the reason you are not seeing an improvement between versions is because you are using a `RAMDirectory`, which bypasses the codecs and some other components that we have improved.
I added benchmarks using the `IndexFiles` and `SearchFiles` from `Lucene.Net.Demo` as templates. These are set up to use a fixed seed to generate the same data over and over again so we don't have to add several megabytes of test files to the repo just to do a benchmark.
Index speed doesn't seem to have improved, but search speed has doubled since 4.8.0-beta00005 (the one with the most downloads on NuGet.org). Here are the results for .NET Core 3.1 on on Windows.
## Indexing
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
```
| Method | Job | NuGetReferences | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|----------- |---------------- |------------------------------------------- |---------:|---------:|---------:|-----------:|----------:|----------:|----------:|
| IndexFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005 | 734.9 ms | 37.14 ms | 55.58 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09 MB |
| IndexFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006 | 703.7 ms | 13.92 ms | 20.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.07 MB |
| IndexFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007 | 705.3 ms | 15.95 ms | 23.38 ms | 43000.0000 | 8000.0000 | 7000.0000 | 221 MB |
| IndexFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008 | 747.0 ms | 42.43 ms | 59.47 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.39 MB |
| IndexFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009 | 725.3 ms | 25.59 ms | 36.70 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.31 MB |
| IndexFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010 | 724.1 ms | 15.41 ms | 22.10 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.23 MB |
## Searching
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
IterationCount=100 LaunchCount=3 WarmupCount=15
```
| Method | Job | NuGetReferences | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------ |---------------- |---------------------------------------------------------------------------------- |----------:|---------:|----------:|----------:|----------:|----------:|------:|----------:|
| SearchFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005,Lucene.Net.QueryParser 4.8.0-beta00005 | 171.18 ms | 2.004 ms | 10.283 ms | 169.51 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006,Lucene.Net.QueryParser 4.8.0-beta00006 | 169.86 ms | 2.666 ms | 13.754 ms | 168.14 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007,Lucene.Net.QueryParser 4.8.0-beta00007 | 173.67 ms | 3.417 ms | 17.595 ms | 169.20 ms | 9000.0000 | 1000.0000 | - | 41.26 MB |
| SearchFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008,Lucene.Net.QueryParser 4.8.0-beta00008 | 95.97 ms | 1.178 ms | 6.068 ms | 95.84 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009,Lucene.Net.QueryParser 4.8.0-beta00009 | 93.31 ms | 0.776 ms | 3.890 ms | 93.60 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010,Lucene.Net.QueryParser 4.8.0-beta00010 | 85.79 ms | 0.307 ms | 1.552 ms | 85.70 ms | 8666.6667 | 500.0000 | - | 40.19 MB |
It would be good to set up a stage on Azure DevOps to run these (off by default) so we can see the results on different operating systems and target frameworks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org