You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "saurabhd336 (via GitHub)" <gi...@apache.org> on 2023/10/03 10:29:31 UTC

[PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

saurabhd336 opened a new pull request, #11729:
URL: https://github.com/apache/pinot/pull/11729

   This PR adds a new interface and it's implementation for existing forward index readers, that allows a caller to get exact set of buffer byte ranges that are accessed internally when reading a given docId. This information can be used to support prefetching of exact byte ranges during query processing if the list of docIds to be fetched is known beforehand, and the buffer type supports selective prefetching of data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349857148


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkForwardIndexReaderV4.java:
##########
@@ -271,10 +302,21 @@ public static abstract class ReaderContext implements ForwardIndexReaderContext
     protected int _nextDocIdOffset;
     protected boolean _regularChunk;
     protected int _numDocsInCurrentChunk;
+    protected long _chunkStartOffset;
+    private List<ValueRange> _ranges;
 
-    protected ReaderContext(PinotDataBuffer metadata, PinotDataBuffer chunks) {
+    protected ReaderContext(PinotDataBuffer metadata, PinotDataBuffer chunks, long chunkStartOffset) {
       _chunks = chunks;
       _metadata = metadata;
+      _chunkStartOffset = chunkStartOffset;
+    }
+
+    public void recordRangesForDocId(int docId, List<ValueRange> ranges) {

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349889744


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for
+     * @return List of {@link ValueRange} that are accessed while reading the given docId
+     */
+    List<ValueRange> getDocIdRange(int docId, T context, @Nullable List<ValueRange> ranges);

Review Comment:
   We actually don't need to.. I had initially thought we'd only allow reading but introduced reusing the list as an after thought optimization. Changed the return type to void



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1356044894


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  // Functions for recording absolute buffer byte ranges accessed while reading a given docId
+
+  /**
+   * Returns whether the forward index supports recording the byte ranges accessed while reading a given docId
+   */
+  default boolean isBufferByteRangeInfoSupported() {

Review Comment:
   Sorry don't follow, why do we need that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348534266


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {

Review Comment:
   It makes sense. Consider it resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1356055209


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  // Functions for recording absolute buffer byte ranges accessed while reading a given docId
+
+  /**
+   * Returns whether the forward index supports recording the byte ranges accessed while reading a given docId
+   */
+  default boolean isBufferByteRangeInfoSupported() {
+    return false;
+  }
+
+  /**
+   * Returns a list of {@link ByteRange} that represents all the distinct

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on PR #11729:
URL: https://github.com/apache/pinot/pull/11729#issuecomment-1756851682

   @Jackie-Jiang I added some code example on how to use these new APIs in the description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348499374


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {

Review Comment:
   What is the reason to do not make all `ForwardIndexReader` extend this method (or just add the methods to `ForwardIndexReader`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348492526


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -326,6 +326,36 @@ private void handlePostCreation()
     }
     updatePostSegmentCreationIndexes(segmentOutputDir);
 
+    Set<IndexType> postSegCreationIndexes = IndexService.getInstance().getAllIndexes().stream()
+        .filter(indexType -> indexType.getIndexBuildLifecycle() == IndexType.BuildLifecycle.POST_SEGMENT_CREATION)
+        .collect(Collectors.toSet());
+
+    if (postSegCreationIndexes.size() > 0) {

Review Comment:
   Wasn't this included in another PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11729:
URL: https://github.com/apache/pinot/pull/11729#issuecomment-1744749931

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11729](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b93826b) into [master](https://app.codecov.io/gh/apache/pinot/commit/e0a1f621b37b6e362dfdb5b7b4c63b72d134f01c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (e0a1f62) will **decrease** coverage by `48.75%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #11729       +/-   ##
   =============================================
   - Coverage     63.13%   14.38%   -48.75%     
   + Complexity     1118      201      -917     
   =============================================
     Files          2342     2342               
     Lines        125883   126179      +296     
     Branches      19357    19387       +30     
   =============================================
   - Hits          79477    18157    -61320     
   - Misses        40749   106482    +65733     
   + Partials       5657     1540     -4117     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.38% <0.00%> (-48.61%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.38% <0.00%> (-48.75%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.38% <0.00%> (-48.75%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.38% <0.00%> (-0.06%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-86.27%)` | :arrow_down: |
   | [.../segment/index/dictionary/DictionaryIndexType.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2RpY3Rpb25hcnkvRGljdGlvbmFyeUluZGV4VHlwZS5qYXZh) | `0.00% <0.00%> (-83.84%)` | :arrow_down: |
   | [.../local/segment/index/forward/ForwardIndexType.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2ZvcndhcmQvRm9yd2FyZEluZGV4VHlwZS5qYXZh) | `0.00% <0.00%> (-91.08%)` | :arrow_down: |
   | [...al/segment/index/nullvalue/NullValueIndexType.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L251bGx2YWx1ZS9OdWxsVmFsdWVJbmRleFR5cGUuamF2YQ==) | `0.00% <0.00%> (-84.00%)` | :arrow_down: |
   | [...gment/index/forward/ForwardIndexReaderFactory.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2ZvcndhcmQvRm9yd2FyZEluZGV4UmVhZGVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-92.00%)` | :arrow_down: |
   | [...ment/index/readers/forward/ChunkReaderContext.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvZm9yd2FyZC9DaHVua1JlYWRlckNvbnRleHQuamF2YQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [.../org/apache/pinot/segment/spi/index/IndexType.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L0luZGV4VHlwZS5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
   | [.../readers/forward/FixedBitSVForwardIndexReader.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvZm9yd2FyZC9GaXhlZEJpdFNWRm9yd2FyZEluZGV4UmVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...eaders/forward/FixedBitSVForwardIndexReaderV2.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvZm9yd2FyZC9GaXhlZEJpdFNWRm9yd2FyZEluZGV4UmVhZGVyVjIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...rs/forward/FixedByteChunkSVForwardIndexReader.java](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvZm9yd2FyZC9GaXhlZEJ5dGVDaHVua1NWRm9yd2FyZEluZGV4UmVhZGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [10 more](https://app.codecov.io/gh/apache/pinot/pull/11729?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   ... and [1496 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11729/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348529553


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {

Review Comment:
   I wanted to avoid having to implement it for other implementations like mutable forward index etc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "zhtaoxiang (via GitHub)" <gi...@apache.org>.
zhtaoxiang commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349349584


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {

Review Comment:
   Constructor, getters and tostring can be simplified using lombok after https://github.com/apache/pinot/pull/11742



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedBytePower2ChunkSVForwardIndexReader.java:
##########
@@ -111,4 +115,43 @@ protected ByteBuffer getChunkBuffer(int docId, ChunkReaderContext context) {
     }
     return decompressChunk(chunkId, context);
   }
+
+  protected void recordDocIdRanges(int docId, ChunkReaderContext context, List<ValueRange> ranges) {
+    int chunkId = docId >>> _shift;
+    if (context.getChunkId() == chunkId) {
+      ranges.addAll(context.getRanges());
+      return;
+    }
+    recordChunkRanges(chunkId, context, ranges);
+  }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    if (_isCompressed) {
+      recordDocIdRanges(docId, context, ranges);
+    } else {
+      // If uncompressed, should use fixed offset
+      throw new IllegalStateException("Operation not supported since the forward index is of fixed length type");
+    }
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    return _rawDataStart;
+  }
+
+  @Override
+  public int getDocLength() {
+    return _storedType.size();
+  }

Review Comment:
   same here: should we throw exception when `isFixedLengthType()` returns false?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java:
##########
@@ -91,4 +95,34 @@ public double getDouble(int docId, ChunkReaderContext context) {
       return _rawData.getDouble(docId * Double.BYTES);
     }
   }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (!_isCompressed) {
+      // If uncompressed, should use fixed offset
+      throw new IllegalStateException("Operation not supported since the forward index is fixed length type");
+    }
+
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    recordDocIdRanges(docId, context, ranges);
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    return _rawDataStart;
+  }
+
+  @Override
+  public int getDocLength() {
+    return _storedType.size();

Review Comment:
   should we throw exception when `isFixedLengthType()` returns false?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/ChunkReaderContext.java:
##########
@@ -56,6 +62,14 @@ public void setChunkId(int chunkId) {
     _chunkId = chunkId;
   }
 
+  public void setRanges(List<ForwardIndexReader.ValueRange> ranges) {

Review Comment:
   similarly, use lombok to simplify code



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for

Review Comment:
   can you please also add java doc for other parameters? currently, we only have java doc for the first parameter



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for
+     * @return List of {@link ValueRange} that are accessed while reading the given docId
+     */
+    List<ValueRange> getDocIdRange(int docId, T context, @Nullable List<ValueRange> ranges);

Review Comment:
   could you please help me understand why we need to return the same type (do we want to change the return type to void?)? IIUC, we want to modify the parameter `List<ValueRange> ranges` directly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349889095


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1353373261


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {
+    private final long _offset;
+    private final int _sizeInBytes;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {

Review Comment:
   Is this useful? This is the same as the constructor



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.
+    private final boolean _isSizeOfBit;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes, false);
+    }
+
+    public static ValueRange newBitRange(long offset, int sizeInBits) {
+      return new ValueRange(offset, sizeInBits, true);
+    }
+
+    private ValueRange(long offset, int size, boolean isSizeOfBit) {
+      _offset = offset;
+      _size = size;
+      _isSizeOfBit = isSizeOfBit;
+    }
+
+    public long getOffset() {
+      return _offset;
+    }
+
+    public int getSize() {
+      return _size;
+    }
+
+    public boolean isSizeOfBit() {
+      return _isSizeOfBit;
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _size + ", _isSizeOfBit=" + _isSizeOfBit + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {

Review Comment:
   I'd also suggest directly adding it into `ForwardIndexReader`. We may add default implementation for the methods, and potentially add `boolean isByteRangeSupported()`



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {

Review Comment:
   Suggest naming it `ByteRange`



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {
+    private final long _offset;
+    private final int _sizeInBytes;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes);
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _sizeInBytes + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for
+     * @param context Reader context
+     * @param ranges List of {@link ValueRange} to which the applicable value ranges will be added
+     */
+    void recordDocIdByteRanges(int docId, T context, List<ValueRange> ranges);

Review Comment:
   Will the caller use this method when the value is fixed length?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1353988984


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {
+    private final long _offset;
+    private final int _sizeInBytes;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {

Review Comment:
   Earlier we had a few variants here. Now it is indeed same as the constructor. Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1354080442


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkMVForwardIndexReader.java:
##########
@@ -195,6 +220,38 @@ private int getValueEndOffset(int rowId, ByteBuffer chunkBuffer) {
     }
   }
 
+  private long getValueEndOffsetAndRecordRanges(int chunkId, int chunkRowId, long chunkStartOffset,

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349154892


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java:
##########
@@ -195,4 +212,61 @@ private long getValueEndOffset(int chunkId, int chunkRowId, long chunkStartOffse
       }
     }
   }
+
+  private long getValueEndOffsetAndRecordRanges(int chunkId, int chunkRowId, long chunkStartOffset,
+      List<ValueRange> ranges) {
+    if (chunkId == _numChunks - 1) {
+      // Last chunk
+      if (chunkRowId == _numDocsPerChunk - 1) {
+        // Last row in the last chunk
+        return _dataBuffer.size();
+      } else {
+        ranges.add(ValueRange.newByteRange(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE, Integer.BYTES));
+        int valueEndOffsetInChunk = _dataBuffer.getInt(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE);
+        if (valueEndOffsetInChunk == 0) {
+          // Last row in the last chunk (chunk is incomplete, which stores 0 as the offset for the absent rows)
+          return _dataBuffer.size();
+        } else {
+          return chunkStartOffset + valueEndOffsetInChunk;
+        }
+      }
+    } else {
+      if (chunkRowId == _numDocsPerChunk - 1) {
+        // Last row in the chunk
+        return getChunkPositionAndRecordRanges(chunkId + 1, ranges);
+      } else {
+        ranges.add(ValueRange.newByteRange(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE, Integer.BYTES));
+        return chunkStartOffset + _dataBuffer.getInt(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE);
+      }
+    }
+  }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    if (_isCompressed) {
+      recordDocIdRanges(docId, context, ranges);
+    } else {
+      recordDocIdRanges(docId, ranges);
+    }
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return false;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    throw new IllegalStateException("Operation not supported since the forward index is not of fixed length type");

Review Comment:
   use `UnsupportedOperationException("Forward index is not of fixed length type")` instead, and many other places as well?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkMVForwardIndexReader.java:
##########
@@ -179,6 +183,20 @@ private byte[] getBytesCompressed(int docId, ChunkReaderContext context) {
     return bytes;
   }
 
+  public void recordDocIdRanges(int docId, List<ValueRange> ranges) {

Review Comment:
   this is a private method? 
   
   btw, changes in this class look quite similar to those in VarByteChunkSVForwardIndexReader.java below, so is there chance to extract common parts to reuse?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkForwardIndexReaderV4.java:
##########
@@ -271,10 +302,21 @@ public static abstract class ReaderContext implements ForwardIndexReaderContext
     protected int _nextDocIdOffset;
     protected boolean _regularChunk;
     protected int _numDocsInCurrentChunk;
+    protected long _chunkStartOffset;
+    private List<ValueRange> _ranges;
 
-    protected ReaderContext(PinotDataBuffer metadata, PinotDataBuffer chunks) {
+    protected ReaderContext(PinotDataBuffer metadata, PinotDataBuffer chunks, long chunkStartOffset) {
       _chunks = chunks;
       _metadata = metadata;
+      _chunkStartOffset = chunkStartOffset;
+    }
+
+    public void recordRangesForDocId(int docId, List<ValueRange> ranges) {

Review Comment:
   private or protected method?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkForwardIndexReaderV4.java:
##########
@@ -330,14 +371,37 @@ private byte[] decompressAndRead(int docId)
       }
       return processChunkAndReadFirstValue(docId, offset, limit);
     }
+
+    private void initAndRecordRangesForDocId(int docId, List<ValueRange> ranges) {
+      // We'd practically end up reading entire metadata buffer

Review Comment:
   expand the comment a bit, to explain why we'd read the entire metadata buffer 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkMVForwardIndexReader.java:
##########
@@ -195,6 +220,38 @@ private int getValueEndOffset(int rowId, ByteBuffer chunkBuffer) {
     }
   }
 
+  private long getValueEndOffsetAndRecordRanges(int chunkId, int chunkRowId, long chunkStartOffset,

Review Comment:
   this method looks similar with other classes, might have a chance to extract common code to reuse



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkForwardIndexReaderV4.java:
##########
@@ -263,6 +270,30 @@ public void close()
       throws IOException {
   }
 
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    context.recordRangesForDocId(docId, ranges);
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return false;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    throw new UnsupportedOperationException();

Review Comment:
   add error msg to be consistent with other places.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349859892


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 merged PR #11729:
URL: https://github.com/apache/pinot/pull/11729


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1354125407


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {
+    private final long _offset;
+    private final int _sizeInBytes;
+
+    public static ValueRange newByteRange(long offset, int sizeInBytes) {
+      return new ValueRange(offset, sizeInBytes);
+    }
+
+    @Override
+    public String toString() {
+      return "Range{" + "_offset=" + _offset + ", _size=" + _sizeInBytes + '}';
+    }
+  }
+
+  /**
+   * An interface that enables the caller to get byte ranges accessed while reading a given docId.
+   * This can be used to eventually allow prefetching of exact byte ranges during query processing,
+   * if the docIds to be fetched are known beforehand.
+   *
+   * This interface will be implemented by all the forward index readers.
+   */
+  interface ValueRangeProvider<T extends ForwardIndexReaderContext> {
+    /**
+     * Returns a list of {@link ValueRange} that represents all the distinct
+     * buffer byte ranges (absolute offset, sizeInBytes) that are accessed when reading the given (@param docId}
+     * @param docId to find the range for
+     * @param context Reader context
+     * @param ranges List of {@link ValueRange} to which the applicable value ranges will be added
+     */
+    void recordDocIdByteRanges(int docId, T context, List<ValueRange> ranges);

Review Comment:
   This method is not supposed to be used if value is fixed length type. Most implementations throw unsupported exception if using this on a fixed offset type fwd index



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1353988772


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +919,67 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * This class represents the buffer byte ranges accessed while reading a given docId.
+   * See {@link ValueRangeProvider} for more details.
+   */
+  @AllArgsConstructor
+  @EqualsAndHashCode
+  @Getter
+  class ValueRange {

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1355756437


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java:
##########
@@ -91,4 +92,41 @@ public double getDouble(int docId, ChunkReaderContext context) {
       return _rawData.getDouble(docId * Double.BYTES);
     }
   }
+
+  @Override
+  public boolean isBufferByteRangeInfoSupported() {
+    return true;
+  }
+
+  @Override
+  public void recordDocIdByteRanges(int docId, ChunkReaderContext context, @Nullable List<ByteRange> ranges) {
+    if (!_isCompressed) {
+      // If uncompressed, should use fixed offset
+      throw new UnsupportedOperationException("Forward index is fixed length type");
+    }
+    recordDocIdRanges(docId, context, ranges);
+  }
+
+  @Override
+  public boolean isFixedOffsetMappingType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getRawDataStartOffset() {
+    if (isFixedOffsetMappingType()) {
+      return _rawDataStart;
+    } else {
+      throw new UnsupportedOperationException("Forward index is not fixed length type");

Review Comment:
   nit: can throw exp w/o `else {}` as if-branch has `return`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348493740


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexReaderFactory.java:
##########
@@ -41,7 +41,15 @@
 
 
 public class ForwardIndexReaderFactory extends IndexReaderFactory.Default<ForwardIndexConfig, ForwardIndexReader> {
-  public static final ForwardIndexReaderFactory INSTANCE = new ForwardIndexReaderFactory();
+  private static ForwardIndexReaderFactory _instance = new ForwardIndexReaderFactory();
+
+  public static void setInstance(ForwardIndexReaderFactory factory) {

Review Comment:
   I think either the variable should be volatile or we should clearly indicate here that the instance can only be changed at startup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348528482


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349861403


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/ChunkReaderContext.java:
##########
@@ -56,6 +62,14 @@ public void setChunkId(int chunkId) {
     _chunkId = chunkId;
   }
 
+  public void setRanges(List<ForwardIndexReader.ValueRange> ranges) {

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349929829


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java:
##########
@@ -91,4 +95,34 @@ public double getDouble(int docId, ChunkReaderContext context) {
       return _rawData.getDouble(docId * Double.BYTES);
     }
   }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (!_isCompressed) {
+      // If uncompressed, should use fixed offset
+      throw new IllegalStateException("Operation not supported since the forward index is fixed length type");
+    }
+
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    recordDocIdRanges(docId, context, ranges);
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    return _rawDataStart;
+  }
+
+  @Override
+  public int getDocLength() {
+    return _storedType.size();

Review Comment:
   Good point. Added



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedBytePower2ChunkSVForwardIndexReader.java:
##########
@@ -111,4 +115,43 @@ protected ByteBuffer getChunkBuffer(int docId, ChunkReaderContext context) {
     }
     return decompressChunk(chunkId, context);
   }
+
+  protected void recordDocIdRanges(int docId, ChunkReaderContext context, List<ValueRange> ranges) {
+    int chunkId = docId >>> _shift;
+    if (context.getChunkId() == chunkId) {
+      ranges.addAll(context.getRanges());
+      return;
+    }
+    recordChunkRanges(chunkId, context, ranges);
+  }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    if (_isCompressed) {
+      recordDocIdRanges(docId, context, ranges);
+    } else {
+      // If uncompressed, should use fixed offset
+      throw new IllegalStateException("Operation not supported since the forward index is of fixed length type");
+    }
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    return _rawDataStart;
+  }
+
+  @Override
+  public int getDocLength() {
+    return _storedType.size();
+  }

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348528233


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexReaderFactory.java:
##########
@@ -41,7 +41,15 @@
 
 
 public class ForwardIndexReaderFactory extends IndexReaderFactory.Default<ForwardIndexConfig, ForwardIndexReader> {
-  public static final ForwardIndexReaderFactory INSTANCE = new ForwardIndexReaderFactory();
+  private static ForwardIndexReaderFactory _instance = new ForwardIndexReaderFactory();
+
+  public static void setInstance(ForwardIndexReaderFactory factory) {

Review Comment:
   Ack will mark as volatile.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1355906002


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  // Functions for recording absolute buffer byte ranges accessed while reading a given docId
+
+  /**
+   * Returns whether the forward index supports recording the byte ranges accessed while reading a given docId
+   */
+  default boolean isBufferByteRangeInfoSupported() {
+    return false;
+  }
+
+  /**
+   * Returns a list of {@link ByteRange} that represents all the distinct

Review Comment:
   Let's add some javadoc stating only one set of the APIs (either fixed length based or record based) should be used



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  // Functions for recording absolute buffer byte ranges accessed while reading a given docId
+
+  /**
+   * Returns whether the forward index supports recording the byte ranges accessed while reading a given docId
+   */
+  default boolean isBufferByteRangeInfoSupported() {

Review Comment:
   Should we consider adding a method to return the underlying `PinotDataBuffer`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1348495585


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java:
##########
@@ -916,4 +916,83 @@ default byte[][] getBytesMV(int docId, T context) {
   default int getNumValuesMV(int docId, T context) {
     throw new UnsupportedOperationException();
   }
+
+  class ValueRange {
+    private final long _offset;
+    private final int _size;
+    // To tell if size uses bit or byte as unit, for fwd index reader reading values of fixed bits.

Review Comment:
   Can we use javadoc here instead of single line comments? It would be useful when reading it from the IDE



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1356044470


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java:
##########
@@ -91,4 +92,41 @@ public double getDouble(int docId, ChunkReaderContext context) {
       return _rawData.getDouble(docId * Double.BYTES);
     }
   }
+
+  @Override
+  public boolean isBufferByteRangeInfoSupported() {
+    return true;
+  }
+
+  @Override
+  public void recordDocIdByteRanges(int docId, ChunkReaderContext context, @Nullable List<ByteRange> ranges) {
+    if (!_isCompressed) {
+      // If uncompressed, should use fixed offset
+      throw new UnsupportedOperationException("Forward index is fixed length type");
+    }
+    recordDocIdRanges(docId, context, ranges);
+  }
+
+  @Override
+  public boolean isFixedOffsetMappingType() {
+    return !_isCompressed;
+  }
+
+  @Override
+  public long getRawDataStartOffset() {
+    if (isFixedOffsetMappingType()) {
+      return _rawDataStart;
+    } else {
+      throw new UnsupportedOperationException("Forward index is not fixed length type");

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349887740


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkForwardIndexReaderV4.java:
##########
@@ -330,14 +371,37 @@ private byte[] decompressAndRead(int docId)
       }
       return processChunkAndReadFirstValue(docId, offset, limit);
     }
+
+    private void initAndRecordRangesForDocId(int docId, List<ValueRange> ranges) {
+      // We'd practically end up reading entire metadata buffer

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #11729:
URL: https://github.com/apache/pinot/pull/11729#discussion_r1349856661


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java:
##########
@@ -195,4 +212,61 @@ private long getValueEndOffset(int chunkId, int chunkRowId, long chunkStartOffse
       }
     }
   }
+
+  private long getValueEndOffsetAndRecordRanges(int chunkId, int chunkRowId, long chunkStartOffset,
+      List<ValueRange> ranges) {
+    if (chunkId == _numChunks - 1) {
+      // Last chunk
+      if (chunkRowId == _numDocsPerChunk - 1) {
+        // Last row in the last chunk
+        return _dataBuffer.size();
+      } else {
+        ranges.add(ValueRange.newByteRange(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE, Integer.BYTES));
+        int valueEndOffsetInChunk = _dataBuffer.getInt(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE);
+        if (valueEndOffsetInChunk == 0) {
+          // Last row in the last chunk (chunk is incomplete, which stores 0 as the offset for the absent rows)
+          return _dataBuffer.size();
+        } else {
+          return chunkStartOffset + valueEndOffsetInChunk;
+        }
+      }
+    } else {
+      if (chunkRowId == _numDocsPerChunk - 1) {
+        // Last row in the chunk
+        return getChunkPositionAndRecordRanges(chunkId + 1, ranges);
+      } else {
+        ranges.add(ValueRange.newByteRange(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE, Integer.BYTES));
+        return chunkStartOffset + _dataBuffer.getInt(chunkStartOffset + (chunkRowId + 1) * ROW_OFFSET_SIZE);
+      }
+    }
+  }
+
+  @Override
+  public List<ValueRange> getDocIdRange(int docId, ChunkReaderContext context, @Nullable List<ValueRange> ranges) {
+    if (ranges == null) {
+      ranges = new ArrayList<>();
+    }
+    if (_isCompressed) {
+      recordDocIdRanges(docId, context, ranges);
+    } else {
+      recordDocIdRanges(docId, ranges);
+    }
+
+    return ranges;
+  }
+
+  @Override
+  public boolean isFixedLengthType() {
+    return false;
+  }
+
+  @Override
+  public long getBaseOffset() {
+    throw new IllegalStateException("Operation not supported since the forward index is not of fixed length type");

Review Comment:
   Ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org