[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5177: Lucene DocId to PinotDocId cache

[GitBox <gi...@apache.org>]

kishoreg commented on a change in pull request #5177: Lucene DocId to PinotDocId cache
URL: https://github.com/apache/incubator-pinot/pull/5177#discussion_r400572893
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
 ##########
 @@ -169,5 +178,57 @@ public void close()
       throws IOException {
     _indexReader.close();
     _indexDirectory.close();
+    _docIdReaderWriter.close();
+  }
+
+  private class DocIdReaderWriter implements Closeable {
+    private PinotDataBuffer _buffer;
+    private final boolean _mappingExists;
+
+    DocIdReaderWriter(File segmentIndexDir, String column, int numDocs)
+        throws Exception {
+      int length = Integer.BYTES * numDocs;
+      File docIdMappingFile = new File(SegmentDirectoryPaths.findSegmentDirectory(segmentIndexDir),
+          column + LUCENE_TEXT_INDEX_DOCID_MAPPING_FILE_EXTENSION);
+      // The mapping is local to a segment. It is created on the server during segment load.
+      // Unless we are running Pinot on Solaris/SPARC, the underlying architecture is
+      // LITTLE_ENDIAN (Linux/x86). So use that as byte order.
+      if (docIdMappingFile.exists()) {
+        // we will be here for segment reload and server restart
+        // for refresh, we will not be here since segment is deleted/replaced
+        // TODO: see if we can prefetch the pages
+        _mappingExists = true;
+        _buffer = PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */ true, 0, length, ByteOrder.LITTLE_ENDIAN,
+            _column + getClass().getSimpleName());
+      } else {
+        _mappingExists = false;
+        _buffer = PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */ false, 0, length, ByteOrder.LITTLE_ENDIAN,
+            _column + getClass().getSimpleName());
+      }
+    }
+
+    public void buildDocIdMapping(int numDocs) {
+      if (!_mappingExists) {
+        for (int i = 0; i < numDocs; i++) {
+          try {
+            Document document = _indexSearcher.doc(i);
+            int pinotDocId = Integer.valueOf(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
+            _buffer.putInt(i * Integer.BYTES, pinotDocId);
+          } catch (Exception e) {
+            throw new RuntimeException("Failed to build doc id mapping during segment load: " + e);
 
 Review comment:
   , e

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org