You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2023/01/04 17:30:37 UTC
[GitHub] [lucene] vstrout opened a new issue, #12067: Getting exception on search after upgrading to Lucene 9.4
vstrout opened a new issue, #12067:
URL: https://github.com/apache/lucene/issues/12067
### Description
After upgrading from Lucene 9.3.0 to Lucene 9.4.2 the index search with sorting by description throws the following exception:
Caused by: java.lang.IllegalStateException: Term [77 73 64 66 6a 66 73 67 73 20 61 64 6b 66 64 6a 68 74 67 64 67 20 61 64 6b 66 64 6a 68 74 67 64 67 20 72 65 74 72 65 72 74 65 20] exists in doc values but not in the terms index
at org.apache.lucene.search.comparators.TermOrdValComparator$CompetitiveIterator.init(TermOrdValComparator.java:582)
at org.apache.lucene.search.comparators.TermOrdValComparator$CompetitiveIterator.update(TermOrdValComparator.java:553)
at org.apache.lucene.search.comparators.TermOrdValComparator$TermOrdValLeafComparator.updateCompetitiveIterator(TermOrdValComparator.java:457)
at org.apache.lucene.search.comparators.TermOrdValComparator$TermOrdValLeafComparator.setHitsThresholdReached(TermOrdValComparator.java:284)
at org.apache.lucene.search.TopFieldCollector$TopFieldLeafCollector.countHit(TopFieldCollector.java:86)
at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$1.collect(TopFieldCollector.java:202)
at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:305)
at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:247)
at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:744)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:662)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:656)
at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:636)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:553)
I wrote a small java function that reproduces the issue. Increasing maxRows value there to above 200 somehow resolves the problem.
public static void test() throws Exception {
String name = "description";
String content = "content";
File dir = new File("C:\\test");
MMapDirectory directory = new MMapDirectory(dir.toPath());
Analyzer analyzer = new StandardAnalyzer(EnglishAnalyzer.ENGLISH_STOP_WORDS_SET);
boolean createIndex = true; // must be changed to false after calling this function first time
if (createIndex) {
String[] values = {
"anshduvfv ",
"dhisdefihfhg ",
"Afasdfasdf ",
"Retrerte ",
"bnbssdfgfg ",
"wrgfhhjg ",
"jfhtvg ",
"fhdhfdsads ",
"Wsdfjfsgs ",
"adkfdjhtgdg ",
};
Random random = new Random();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
config.setOpenMode(OpenMode.CREATE_OR_APPEND);
IndexWriter writer = new IndexWriter(directory, config);
for (int i = 0; i < 110; i++) {
for (int j = 0; j < 10; j++) {
String value = values[j] + values[random.nextInt(10)] + values[random.nextInt(10)] + values[random.nextInt(10)];
Document doc = new Document();
doc.add(new TextField(content, value, Field.Store.NO));
doc.add(new StringField(name, value, Field.Store.YES));
doc.add(new SortedDocValuesField(name, new BytesRef(value.toLowerCase()))); // case-insensitive sorting
writer.addDocument(doc);
}
}
writer.close();
}
int maxRows = 100;
String request = "*:*";
DirectoryReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);
QueryParser parser = new QueryParser(content, analyzer);
parser.setSplitOnWhitespace(true);
parser.setAllowLeadingWildcard(false);
Query query = parser.parse(request);
Sort sort = new Sort(new SortField(name, Type.STRING, true));
TopDocs docs = searcher.search(query, maxRows, sort);
reader.close();
}
### Version and environment details
Upgraded from Lucene 9.3.0 to Lucene 9.4.2. Using lucene-backward-codecs-9.4.2.jar
OS: MS Windows 11
Java: jdk-11.0.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12067: Getting exception on search after upgrading to Lucene 9.4
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12067:
URL: https://github.com/apache/lucene/issues/12067#issuecomment-1374409314
The bug is in your code, because you index inconsitently different data in the same field: the data going into docvalues is lowercased, the data going into stringfield is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12067: Getting exception on search after upgrading to Lucene 9.4
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12067:
URL: https://github.com/apache/lucene/issues/12067#issuecomment-1377207063
@vstrout You may also use this setter as a quick-fix: https://github.com/apache/lucene/blob/branch_9x/lucene/core/src/java/org/apache/lucene/search/SortField.java#L636-L657
But long term, again I would recommend just naming the sort fields differently, if they have different content (e.g. `description_lowercase` or `description_sort` or whatever you want, anything but `description` since the contents are in fact different)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12067: Getting exception on search after upgrading to Lucene 9.4
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12067:
URL: https://github.com/apache/lucene/issues/12067#issuecomment-1377199993
If you use a different field name for the lowercased version, then you won't have an issue. For example `description_lowercase`.
The problem is that you call it `description` so lucene expects it to be consistent. This is for performance reasons, so it can use all the datastructures available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] vstrout commented on issue #12067: Getting exception on search after upgrading to Lucene 9.4
Posted by GitBox <gi...@apache.org>.
vstrout commented on issue #12067:
URL: https://github.com/apache/lucene/issues/12067#issuecomment-1375885952
Do you mind to point out what will be a proper solution then for case-insensitive sorting? The code from my example was working for us with all versions of Lucene prior to 9.4. Appreciate your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir closed issue #12067: Getting exception on search after upgrading to Lucene 9.4
Posted by GitBox <gi...@apache.org>.
rmuir closed issue #12067: Getting exception on search after upgrading to Lucene 9.4
URL: https://github.com/apache/lucene/issues/12067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org