You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by GitBox <gi...@apache.org> on 2019/02/10 09:48:18 UTC
[GitHub] jainankitk commented on a change in pull request #563: Support for
moving FST offheap except PK
jainankitk commented on a change in pull request #563: Support for moving FST offheap except PK
URL: https://github.com/apache/lucene-solr/pull/563#discussion_r255330316
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/blocktree/FieldReader.java
##########
@@ -86,9 +88,14 @@
if (indexIn != null) {
final IndexInput clone = indexIn.clone();
- //System.out.println("start=" + indexStartFP + " field=" + fieldInfo.name);
clone.seek(indexStartFP);
- index = new FST<>(clone, ByteSequenceOutputs.getSingleton());
+ // Initialize FST offheap if index is MMapDirectory and
+ // docCount != sumDocFreq implying field is not primary key
+ if (clone instanceof ByteBufferIndexInput && this.docCount != this.sumDocFreq) {
+ index = new FST<>(clone, ByteSequenceOutputs.getSingleton(), new OffHeapFSTStore());
+ } else {
+ index = new FST<>(clone, ByteSequenceOutputs.getSingleton());
Review comment:
I added print statements while running the benchmarks, and the classification looks correct:
```
Initializing field offheap start=55 field=Date.taxonomy
Initializing field offheap start=76 field=DayOfYear.sortedset
Initializing field offheap start=97 field=Month.sortedset
Initializing field offheap start=118 field=body
Initializing field onheap start=267 field=date
Initializing field onheap start=289 field=groupend
Initializing field onheap start=311 field=id
Initializing field onheap start=333 field=title
```
Though, when I restricted tests to PKLookups only using comp.addTaskPattern('PKLookup') in localrun.py, results look as expected:
```
wikimedium10k
TaskQPS baseline StdDevQPS candidate StdDev Pct diff
PKLookup 163.29 (1.6%) 164.80 (2.1%) 0.9% (-2% - 4%)
```
```
wikimedium10m
TaskQPS baseline StdDevQPS candidate StdDev Pct diff
PKLookup 114.29 (1.7%) 114.73 (1.2%) 0.4% ( -2% - 3%)
```
I guess we are good then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org