You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/04/15 14:44:25 UTC

[GitHub] [lucene] iverase opened a new pull request #85: LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

iverase opened a new pull request #85:
URL: https://github.com/apache/lucene/pull/85


    `MonotonicBlockPackedReader` and `MonotonicBlockPackedWriter`  are used in the `FixedGapTermsIndexReader` and `FixedGapTermsIndexWriter` respectively. Those readers/ writers from internally from PackedInts#getReader. This PR removes that dependency by using internally DirectReaders and DirectWriters instead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] jpountz commented on a change in pull request #85: LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #85:
URL: https://github.com/apache/lucene/pull/85#discussion_r615246017



##########
File path: lucene/core/src/java/org/apache/lucene/util/packed/MonotonicBlockPackedReader.java
##########
@@ -72,24 +75,44 @@ private MonotonicBlockPackedReader(
         throw new IOException("Corrupted");
       }
       if (bitsPerValue == 0) {
-        subReaders[i] = new PackedInts.NullReader(blockSize);
+        subReaders[i] = LongValues.ZEROES;
       } else {
         final int size = (int) Math.min(blockSize, valueCount - (long) i * blockSize);
-        if (direct) {
-          final long pointer = in.getFilePointer();
-          subReaders[i] =
-              PackedInts.getDirectReaderNoHeader(
-                  in, PackedInts.Format.PACKED, packedIntsVersion, size, bitsPerValue);
-          in.seek(
-              pointer + PackedInts.Format.PACKED.byteCount(packedIntsVersion, size, bitsPerValue));
-        } else {
-          subReaders[i] =
-              PackedInts.getReaderNoHeader(
-                  in, PackedInts.Format.PACKED, packedIntsVersion, size, bitsPerValue);
-        }
+        blocksSize += size;

Review comment:
       you seem to be using blocksSize to indicate RAM usage, ie. number of bytes in memory, but `size` is actually a number of values? should we add byteCount rather?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase commented on pull request #85: LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

Posted by GitBox <gi...@apache.org>.
iverase commented on pull request #85:
URL: https://github.com/apache/lucene/pull/85#issuecomment-821781482


   Thanks @jpountz, I did not realise about that constraint.
   
   I took a different approach: Similarly to `Packed64`, I read the data into a backing byte array and implement a get method that works directly on that array. `MonotonicBlockPackedReader` seems to be always used on heap (`direct = false`), therefore I remove the flag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase merged pull request #85: LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

Posted by GitBox <gi...@apache.org>.
iverase merged pull request #85:
URL: https://github.com/apache/lucene/pull/85


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase commented on a change in pull request #85: LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

Posted by GitBox <gi...@apache.org>.
iverase commented on a change in pull request #85:
URL: https://github.com/apache/lucene/pull/85#discussion_r615547697



##########
File path: lucene/core/src/java/org/apache/lucene/util/packed/MonotonicBlockPackedReader.java
##########
@@ -72,24 +75,44 @@ private MonotonicBlockPackedReader(
         throw new IOException("Corrupted");
       }
       if (bitsPerValue == 0) {
-        subReaders[i] = new PackedInts.NullReader(blockSize);
+        subReaders[i] = LongValues.ZEROES;
       } else {
         final int size = (int) Math.min(blockSize, valueCount - (long) i * blockSize);
-        if (direct) {
-          final long pointer = in.getFilePointer();
-          subReaders[i] =
-              PackedInts.getDirectReaderNoHeader(
-                  in, PackedInts.Format.PACKED, packedIntsVersion, size, bitsPerValue);
-          in.seek(
-              pointer + PackedInts.Format.PACKED.byteCount(packedIntsVersion, size, bitsPerValue));
-        } else {
-          subReaders[i] =
-              PackedInts.getReaderNoHeader(
-                  in, PackedInts.Format.PACKED, packedIntsVersion, size, bitsPerValue);
-        }
+        blocksSize += size;

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org