You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "iverase (via GitHub)" <gi...@apache.org> on 2023/09/21 10:19:44 UTC

[GitHub] [lucene] iverase opened a new pull request, #12581: Allow reading / writing binary stored values as DataInput

iverase opened a new pull request, #12581:
URL: https://github.com/apache/lucene/pull/12581

   Currently, the only way to handle binary data on stored fields is via byte arrays (wrapped as BytesRef). THis means we are allocating a new byte array everytime we read the value which is wasteful and it can be problematic as those arrays can be humongous and add pressure to the GC. 
   
   In this PR we proposed to add the possibility to read / write binary stored values using a DataInput and the number of bytes. By default the implementations will allocate those bytes in a newly created byte array and call the already existing method. 
   
   This should speed up the merging of stored fields as we are not using an intermediate data structure any longer and allow implementoirs to read the binary fields without having to allocate a byte array.
   
   closes https://github.com/apache/lucene/issues/12556 
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] romseygeek commented on a diff in pull request #12581: Allow reading / writing binary stored fields as DataInput

Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on code in PR #12581:
URL: https://github.com/apache/lucene/pull/12581#discussion_r1333208421


##########
lucene/core/src/java/org/apache/lucene/codecs/StoredFieldsWriter.java:
##########
@@ -182,6 +190,11 @@ public MergeVisitor(MergeState mergeState, int readerIndex) {
       }
     }
 
+    @Override
+    public void binaryField(FieldInfo fieldInfo, DataInput value, int length) throws IOException {
+      writeField(remap(fieldInfo), value, length);
+    }
+
     @Override
     public void binaryField(FieldInfo fieldInfo, byte[] value) throws IOException {
       // TODO: can we avoid new BR here?

Review Comment:
   Does this ever get called now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase merged pull request #12581: Allow reading / writing binary stored fields as DataInput

Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase merged PR #12581:
URL: https://github.com/apache/lucene/pull/12581


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase commented on a diff in pull request #12581: Allow reading / writing binary stored fields as DataInput

Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #12581:
URL: https://github.com/apache/lucene/pull/12581#discussion_r1333415368


##########
lucene/core/src/java/org/apache/lucene/codecs/StoredFieldsWriter.java:
##########
@@ -182,6 +190,11 @@ public MergeVisitor(MergeState mergeState, int readerIndex) {
       }
     }
 
+    @Override
+    public void binaryField(FieldInfo fieldInfo, DataInput value, int length) throws IOException {
+      writeField(remap(fieldInfo), value, length);
+    }
+
     @Override
     public void binaryField(FieldInfo fieldInfo, byte[] value) throws IOException {
       // TODO: can we avoid new BR here?

Review Comment:
   Actually the `SimpleTextCodec` codec is using it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] iverase commented on a diff in pull request #12581: Allow reading / writing binary stored fields as DataInput

Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #12581:
URL: https://github.com/apache/lucene/pull/12581#discussion_r1333235616


##########
lucene/core/src/java/org/apache/lucene/codecs/StoredFieldsWriter.java:
##########
@@ -182,6 +190,11 @@ public MergeVisitor(MergeState mergeState, int readerIndex) {
       }
     }
 
+    @Override
+    public void binaryField(FieldInfo fieldInfo, DataInput value, int length) throws IOException {
+      writeField(remap(fieldInfo), value, length);
+    }
+
     @Override
     public void binaryField(FieldInfo fieldInfo, byte[] value) throws IOException {
       // TODO: can we avoid new BR here?

Review Comment:
   Nope, it should not be called, maybe we should assert that it doesn't get called?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org