You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/03/11 08:52:24 UTC
[GitHub] [lucene-solr] jpountz opened a new pull request #1338: Move
BufferedIndexInput to the ByteBuffer API
jpountz opened a new pull request #1338: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338
See https://issues.apache.org/jira/browse/LUCENE-9271.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1338:
LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#discussion_r390830056
##########
File path: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java
##########
@@ -154,65 +144,60 @@ public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) th
// this function, there is no need to do a seek
// here, because there's no need to reread what we
// had in the buffer.
- long after = bufferStart+bufferPosition+len;
+ long after = bufferStart+buffer.position()+len;
if(after > length())
throw new EOFException("read past EOF: " + this);
- readInternal(b, offset, len);
+ readInternal(ByteBuffer.wrap(b, offset, len));
bufferStart = after;
- bufferPosition = 0;
- bufferLength = 0; // trigger refill() on read
+ buffer.position(0);
+ buffer.limit(0); // trigger refill() on read
}
}
}
@Override
public final short readShort() throws IOException {
- if (2 <= (bufferLength-bufferPosition)) {
- return (short) (((buffer[bufferPosition++] & 0xFF) << 8) | (buffer[bufferPosition++] & 0xFF));
+ if (Short.BYTES <= buffer.remaining()) {
+ return buffer.getShort();
Review comment:
+1. I checked the limit/remaining and other calls but the correctness there seems to be better verified by tests than by hand. :) Overall, like I said - looks like a nice simplification to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1338:
LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#discussion_r390828520
##########
File path: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java
##########
@@ -154,65 +144,60 @@ public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) th
// this function, there is no need to do a seek
// here, because there's no need to reread what we
// had in the buffer.
- long after = bufferStart+bufferPosition+len;
+ long after = bufferStart+buffer.position()+len;
if(after > length())
throw new EOFException("read past EOF: " + this);
- readInternal(b, offset, len);
+ readInternal(ByteBuffer.wrap(b, offset, len));
bufferStart = after;
- bufferPosition = 0;
- bufferLength = 0; // trigger refill() on read
+ buffer.position(0);
+ buffer.limit(0); // trigger refill() on read
}
}
}
@Override
public final short readShort() throws IOException {
- if (2 <= (bufferLength-bufferPosition)) {
- return (short) (((buffer[bufferPosition++] & 0xFF) << 8) | (buffer[bufferPosition++] & 0xFF));
+ if (Short.BYTES <= buffer.remaining()) {
+ return buffer.getShort();
Review comment:
Yes: "Specific byte orders are represented by instances of the ByteOrder class. The initial order of a byte buffer is always BIG_ENDIAN." (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/ByteBuffer.html)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on issue #1338: LUCENE-9271: Move
BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#issuecomment-597619174
Or maybe even simpler: just change `protected` to `private`. I'm not sure if any subclasses really need to mess with the buffer in that way anyway? If there is really a use-case, then we can try to introduce more sanity checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1338:
LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#discussion_r390823155
##########
File path: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java
##########
@@ -154,65 +144,60 @@ public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) th
// this function, there is no need to do a seek
// here, because there's no need to reread what we
// had in the buffer.
- long after = bufferStart+bufferPosition+len;
+ long after = bufferStart+buffer.position()+len;
if(after > length())
throw new EOFException("read past EOF: " + this);
- readInternal(b, offset, len);
+ readInternal(ByteBuffer.wrap(b, offset, len));
bufferStart = after;
- bufferPosition = 0;
- bufferLength = 0; // trigger refill() on read
+ buffer.position(0);
+ buffer.limit(0); // trigger refill() on read
}
}
}
@Override
public final short readShort() throws IOException {
- if (2 <= (bufferLength-bufferPosition)) {
- return (short) (((buffer[bufferPosition++] & 0xFF) << 8) | (buffer[bufferPosition++] & 0xFF));
+ if (Short.BYTES <= buffer.remaining()) {
+ return buffer.getShort();
Review comment:
Can't remember off the top of my head - the byte order is always guaranteed here, right? Or should we assert the buffer has the right one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on issue #1338: LUCENE-9271: Move
BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
dweiss commented on issue #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#issuecomment-597517675
This looks like a very nice simplification to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz closed pull request #1338: LUCENE-9271: Move
BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
jpountz closed pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi commented on a change in pull request #1338:
LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
danmuzi commented on a change in pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#discussion_r391152040
##########
File path: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java
##########
@@ -154,65 +146,60 @@ public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) th
// this function, there is no need to do a seek
// here, because there's no need to reread what we
// had in the buffer.
- long after = bufferStart+bufferPosition+len;
+ long after = bufferStart+buffer.position()+len;
if(after > length())
throw new EOFException("read past EOF: " + this);
- readInternal(b, offset, len);
+ readInternal(ByteBuffer.wrap(b, offset, len));
bufferStart = after;
- bufferPosition = 0;
- bufferLength = 0; // trigger refill() on read
+ buffer.position(0);
Review comment:
I think if there is `buffer.limit(0)` on the next line, `buffer.position(0)` doesn't look necessary.
Because there is a code of initializing position value within `buffer.limit(0)`.
```java
// ByteBuffer.java
public abstract class ByteBuffer extends Buffer implements Comparable<ByteBuffer>
{
...
ByteBuffer limit(int newLimit) {
super.limit(newLimit);
return this;
}
...
}
// Buffer.java
public abstract class Buffer {
...
public Buffer limit(int newLimit) {
if (newLimit > capacity | newLimit < 0)
throw createLimitException(newLimit);
limit = newLimit;
if (position > limit) position = limit; // position will be 0.
if (mark > limit) mark = -1;
return this;
}
...
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on issue #1338: LUCENE-9271: Move
BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
jpountz commented on issue #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#issuecomment-597688851
Good call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on issue #1338: LUCENE-9271: Move
BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#issuecomment-597617523
If we are replacing the `byte[]` with `ByteBuffer` here, can we try to add some safety at the same time? Can the actual buffer be `private final` and only passed to `readInternal()`? Otherwise, subclasses should only be able to see a read-only view (`.asReadOnlyBuffer`). The little bit of safety should be free, and since we are changing the API anyway, we might as well try to do it now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1338:
LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1338: LUCENE-9271: Move BufferedIndexInput to the ByteBuffer API
URL: https://github.com/apache/lucene-solr/pull/1338#discussion_r390828914
##########
File path: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java
##########
@@ -154,65 +144,60 @@ public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) th
// this function, there is no need to do a seek
// here, because there's no need to reread what we
// had in the buffer.
- long after = bufferStart+bufferPosition+len;
+ long after = bufferStart+buffer.position()+len;
if(after > length())
throw new EOFException("read past EOF: " + this);
- readInternal(b, offset, len);
+ readInternal(ByteBuffer.wrap(b, offset, len));
bufferStart = after;
- bufferPosition = 0;
- bufferLength = 0; // trigger refill() on read
+ buffer.position(0);
+ buffer.limit(0); // trigger refill() on read
}
}
}
@Override
public final short readShort() throws IOException {
- if (2 <= (bufferLength-bufferPosition)) {
- return (short) (((buffer[bufferPosition++] & 0xFF) << 8) | (buffer[bufferPosition++] & 0xFF));
+ if (Short.BYTES <= buffer.remaining()) {
+ return buffer.getShort();
Review comment:
I think it's still a good call to add assertions to make sure that sub classes don't mess up with the ByteBuffer. I'll add some.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org