Search degradation on Windows when upgrading from lucene 3.6 to lucene 4.7.2

[Shlomit Rosen <SH...@il.ibm.com>]

Hi, 

We are in the process of upgrading  from lucene 3.6.0 to lucene 4.7.2, 
and our tests show a significant search degradation on Windows platform.

Trying to figure this out, here are a couple of points we noticed. 
Any suggestions/thoughts will be greatly appreciated. 

Thanks!

1) Running search on an optimized collection.

Our first run on Windows machine showed the following results: 
        Lucene 3.6:             115 queries / sec
        Lucene 4.7.2:           74   queries / sec

Looking at the collections themselves, we got the following 
characterization: 


Lucene 3.6
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 116
Total size of files in FOLDER: 81558862032 bytes (75.96 GB)

Commit Point Information:
=========================
Version: 1399567203042
Timestamp: 1399593668185
Generation: 6018
Segments file name: segments_4n6
Number of segments: 32
Committed size: 81216915273 bytes (75.64 GB)
Number of files in COMMIT POINT: 89
Total size of files in COMMIT POINT: 81216923390 bytes (75.64 GB)


Lucene 4.7.2: 
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 301
Total size of files in FOLDER: 71019073768 bytes (66.14 GB)

Commit Point Information:
=========================
Generation: 4518
Segments file name: segments_3hi
Number of segments: 38
Committed size: 70635339707 bytes (65.78 GB)
Number of files in COMMIT POINT: 115
Total size of files in COMMIT POINT: 70635341223 bytes (65.78 GB)

We saw that the collection created by lucene 4.7.2 was10GB smaller but it 
had a more segments. 
We thought that more segments might account to the search degradation, and 
so we decided to run optimization on the 4.7.2 index before rerunning the 
search test. 

The index was more compact: 

Lucene 4.7.2
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 38
Total size of files in FOLDER: 70488334388 bytes (65.65 GB)

Commit Point Information:
=========================
Generation: 4519
Segments file name: segments_3hj
Number of segments: 12
Committed size: 70488333864 bytes (65.65 GB)
Number of files in COMMIT POINT: 37
Total size of files in COMMIT POINT: 70488334368 bytes (65.65 GB)
And as expected, the search results were much better: 
        4.7.2.           118 queries / sec


We thought that this might be a good direction, so our next step was to 
simulate a more compact index as part of our indexing session without 
running a full optimize at the end. 
To do that we changed maxMergeMB from 4 GB to 6 GB. The collection was 
indeed more compact: 


Win64 4.7.2 merge=6000 commitPoints:
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 213
Total size of files in FOLDER: 83038952682 bytes (77.34 GB)

Commit Point Information:
=========================
Generation: 4406
Segments file name: segments_3ee
Number of segments: 14
Committed size: 70324985193 bytes (65.50 GB)
Number of files in COMMIT POINT: 91
Total size of files in COMMIT POINT: 70324985781 bytes (65.50 GB)
But search results were not good at all: 
        4.7.2:          72 queries / sec

Does this make sense? 
We thought of "Optimize" as mainly decreasing the number of segments in 
the collection, and removing deletions. 
In this scenario, we had no deletions, and we saw that the number of 
segments did in fact decrease substantially, 
So why are we not seeing this reflect in search performance? Is there any 
other "optimize" influence/hidden-operation that we are missing here? 

(Note that we are using LogByteSizeMergePolicy. We know that 
TieredMergePolicy is suppose to be better in this aspect, but it is 
important to us 
To keep the order of the documents the same between commit points... )


2) Search Directory
On Lucene 3.6, we did comprehensive testing and saw that the best search 
performance is reached when using an Mmap directory. 
(for Indexing we are using SimpleFSDirectory). 
We tried different directories again with lucene 4.7.2, and while the 
differences were not big, it still seems that Mmap is no longer the best 
option: 

Lucene 4.7.2 with MMap:         72 queries / sec
Lucene 4.7.2 with SimpleFS:     84 queries / sec

Was there any changes around the MMap directory that might account for 
this difference? 
If so, do you think that those changes might account for the overall 
performance we are seeing? 

3) Java 6 / Java 7
We are currently running on Java 6 (that is also the reason we stopped at 
lucene 4.7.2 and not 4.8). 
Is there a reason to believe that the degradation might be connected to 
this? 


Thanks again in advance!