You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "atris (via GitHub)" <gi...@apache.org> on 2023/07/06 20:23:00 UTC
[GitHub] [pinot] atris opened a new pull request, #11046: [Bugfix] WIP -- Use PinotDataBuffer For ImmutableFST
atris opened a new pull request, #11046:
URL: https://github.com/apache/pinot/pull/11046
Use PinotDataBuffer instead of MutableBytesStore for building ImmutableFST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a diff in pull request #11046: [Bugfix] WIP -- Use PinotDataBuffer For ImmutableFST
Posted by "atris (via GitHub)" <gi...@apache.org>.
atris commented on code in PR #11046:
URL: https://github.com/apache/pinot/pull/11046#discussion_r1254885563
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/ImmutableFST.java:
##########
@@ -139,7 +140,11 @@ public final class ImmutableFST extends FST {
* see the documentation of this class for more information on how this
* structure is organized.
*/
- public final OffHeapMutableBytesStore _mutableBytesStore;
+ //public final OffHeapMutableBytesStore _mutableBytesStore;
+
+ public final PinotDataBuffer _pinotDataBuffer;
Review Comment:
@Jackie-Jiang I think I need to use the offset buffer and value buffer approach here instead of incrementing the offset by PER_BUFFER_SIZE each time. Is that correct? (wanted to confirm with you before I make the change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11046: [Bugfix] WIP -- Use PinotDataBuffer For ImmutableFST
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11046:
URL: https://github.com/apache/pinot/pull/11046#discussion_r1259342992
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/NativeFSTIndexReader.java:
##########
@@ -48,8 +51,19 @@ public class NativeFSTIndexReader implements TextIndexReader {
public NativeFSTIndexReader(PinotDataBuffer dataBuffer)
throws IOException {
// TODO: Implement an InputStream directly on PinotDataBuffer
- ByteBuffer byteBuffer = dataBuffer.toDirectByteBuffer(0, (int) dataBuffer.size());
- _fst = FST.read(new ByteBufferInputStream(Collections.singletonList(byteBuffer)), ImmutableFST.class, true);
+ /*ByteBuffer byteBuffer = dataBuffer.toDirectByteBuffer(0, (int) dataBuffer.size());
Review Comment:
We already have `PinotDataBuffer` here, so we should not create an input stream from it, but directly using it to read data. For immutable index, reading from stream is very inefficient because it requires us to copy the data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org