Lucene normal search function not working with tokenize=false settings

["Balekundri, Manjunath V" <ma...@cgi.com>]

Hi!

I need some clarification about how to make Lucene search function to make it work with tokenize=false settings.
Our expectation normal search should work even if we set Tokenize parameter as false.

I am sending this mail along with my java code(LuceneQueryTest.java class: attached) calling Lucene code.


A.      Code structure:
Attached sample code(LuceneQueryTest.java class) consists of 2 steps:

1.       Create index (Needed to be executed only once to insert records)

2.       Process queries



B.      Analysis Report:

Step 1 can be executed only once because it creates index files first time and then it can be commented.

Step 2 has 3 cases

Case a) queryParams = " LocationCode:1"; (To see all records)

                Test analysis: When we execute this query all records are fetched, works fine. (Works Fine. To show all records)

               [Description: cid:image004.png@01D32896.923E7AD0]



Case b) queryParams = "LocationCode:1 AND Category:Rohit Mohit"";

                Test analysis: When we execute this query no records are fetched. Expected two records(1st and 2nd to be shown) (Not working)

[Description: cid:image005.png@01D32896.923E7AD0]



Case c) queryParams = "LocationCode:1 AND Category:a*"; (Not working)

                Test analysis: When we execute this query only one record found. Expected two records (1 & 2 above) should be shown.

               [Description: cid:image006.png@01D32896.923E7AD0]

C.      Submission to lucene development team:

Looking at case b) and case c) I think there is bug in Lucene IndexSearcher.java or IndexReader.java in LuceneCore module.

I used both Lucene 5.3 to test above scenarios.





D.      Some Other Options Tried:



A)     To test with org<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg>.apache<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache>.lucene<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene>.document<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document>.Field<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document(Field.class%E2%98%83Field>

Code:

[cid:image003.png@01D32898.88153E70]



Results:

[cid:image007.png@01D32898.88153E70]



B)      To test with org<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg>.apache<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache>.lucene<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene>.document<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document>.TextField<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document(TextField.class%E2%98%83TextField> and org<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg>.apache<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache>.lucene<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene>.document<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document>.StringField<eclipse-javadoc:%E2%98%82=ArchiveCore/C:%5C/D%5C/ARP%5C/repositoryAll%5C/org%5C/apache%5C/lucene%5C/lucene-core%5C/5.3.0%5C/lucene-core-5.3.0.jar%3Corg.apache.lucene.document(StringField.class%E2%98%83StringField>

Code:

[Description: cid:image001.png@01D32896.DFA7ADF0]

Results:

[Description: cid:image002.png@01D32896.DFA7ADF0]



If you need more information, please let me know. I need to fix these issues in customer release war files.


Best regards,
Rohit