You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dweiss (via GitHub)" <gi...@apache.org> on 2024/09/25 19:12:22 UTC
[PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
dweiss opened a new pull request, #13825:
URL: https://github.com/apache/lucene/pull/13825
Fixes #13820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "easyice (via GitHub)" <gi...@apache.org>.
easyice commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2380612807
Thank you @rmuir, Maybe we can move `readGroupVInts()` into `GroupVIntUtil` to achieve the same goal? I raised a PR #13830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2401573279
I've backported this to branch_10x and asked the RM to allow this to be cherry picked for the subsequent respin of Lucene 10.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2401799763
I think we are also missing a CHANGES entry , @dweiss could you take care of adding that please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "easyice (via GitHub)" <gi...@apache.org>.
easyice commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2378508188
> I understand what the intention is here, @easyice , but this is the problem - you can't override that public method anymore and you can't call the protected method in a delegate class. I sort of agree with Robert that code duplication here is not worth the added API complexity. I suggest we remove the final modifier from that method first, then think of maybe dropping the protected method altogether (although this doesn't bother me as much, since it's protected).
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2378011284
> Yes, this would also solve the problem. I wonder if it makes sense to expose that protected method though, if it's internal details of the implementation? I'd rather make the primary API method public and let subclasses do whatever they want with it (delegate, reimplement, etc.)
I don't understand why there are 2 methods in the first place. There's a little bit of docs on the `final` one, but the `protected` one is basically undocumented. Seems a bit convoluted.
IMO one single, well-documented, public method would be a better fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2375369876
Isn't the root cause of the issue the `protected` method? The `final` method just invokes other methods, so alternatively we could fix the visibility of the protected method to be public like all the others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2379010887
the current readGroupVInts is also pretty specific to what the postings are doing. And it just calls readVint() and readGroupVint() behind the scenes. Seems like a good candidate to move to PostingsUtil, leaving just the one readGroupVint() method in DataInput?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2379139038
i dont think we should add methods to DataInput when we are only "trying to use elsewhere". The method is only used from one place: PostingsUtil, lets move it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2402464116
Thanks @dweiss !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2377881935
> Isn't the root cause of the issue the `protected` method? The `final` method just invokes other methods, so alternatively we could fix the visibility of the protected method to be public like all the others?
Yes, this would also solve the problem. I wonder if it makes sense to expose that protected method though, if it's internal details of the implementation? I'd rather make the primary API method public and let subclasses do whatever they want with it (delegate, reimplement, etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2380624362
> Thank you @rmuir, Maybe we can move `readGroupVInts()` into `GroupVIntUtil` to achieve the same goal? I raised a PR #13830
This is much better. Let's keep only the inner loop part in DataInput and replace the protected method by a public method called by the static utility reading the whole think.
Anyways: I was a bit against the whole change from beginning because I was not happy to add new methods to DataInput just for a silly optimization. I am also not happy about the horrible bottom-up code just to get MemorySegment access working.
IMHO, we should move minimum Java version of Lucene main branch (and 10.x) to Java 22 and introduce MemorySegment as a first class citizen and remove all abstractions from DataInput. Just make DataInput read from/to MemorySegment (NIOFSDir) or let it provide "views" (MMap) on the internal buffers. Let's move all our DataInput encoding/decoding logic to work on top of MemorySegment.
That's all I have to say. I don't care about those arguments "wuhaaah, but Java 21 is LTS !!!!1!1111!!!!!!!!!!". Don't care, use Java 22 because this is what we need to be ready for future.
Uwe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2379761858
Folks, if you'd like to do anything larger to this, please go ahead. I'm with very limited access to the internet this week and I won't be able to track your comments closely or make any code modifications. I'll return next Monday and then can jump back in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "easyice (via GitHub)" <gi...@apache.org>.
easyice commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2379114801
> the current readGroupVInts is also pretty specific to what the postings are doing. And it just calls readVint() and readGroupVint() behind the scenes. Seems like a good candidate to move to PostingsUtil, leaving just the one readGroupVint() method in DataInput?
Actually, I have tried to use group-varint encoding in other places. just don’t see a significant impact on performance at this point.
* Use group-varint encode the positions https://github.com/apache/lucene/pull/12842
* Cut over HNSW's neighbor lists to group-varint https://github.com/apache/lucene/issues/12871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2378477884
I understand what the intention is here, @easyice , but this is the problem - you can't override that public method anymore and you can't call the protected method in a delegate class. I sort of agree with Robert that code duplication here is not worth the added API complexity. I suggest we remove the final modifier from that method first, then think of maybe dropping the protected method altogether (although this doesn't bother me as much, since it's protected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "easyice (via GitHub)" <gi...@apache.org>.
easyice commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2378380489
Sorry for the confusion. We move group-varint encoding into DataInput to let subclasses make an optimized implementation.
In the first version, the `readGroupVInts` method is `public`. see [delta](https://github.com/apache/lucene/pull/12841/commits/e58154e44a62235450b4f9d837e9307f4f5a8619%0ADataInput.java):
class DataInput
```java
public void readGroupVInts(long[] docs, int limit) throws IOException {
fallbackReadGroupVInt(...)
}
```
```java
protected void fallbackReadGroupVInt(long[] docs, int offset) throws IOException {
}
```
subclass MemorySegmentIndexInput
```java
@Override
public void readGroupVInts(long[] docs, int limit) throws IOException {
}
private void readGroupVInt(long[] docs, int pos) throws IOException {
if (...) {
super.fallbackReadGroupVInt(docs, pos);
return;
}
....
}
```
The problem with this version is that subclasses have some duplicate code with small changes. then we try to move the duplicate code to a util class in the subclasses, but it has a bit performance regression. see [delta](https://github.com/apache/lucene/pull/12841#issuecomment-1850424855)
class MemorySegmentIndexInput
```java
@Override
public void readGroupVInts(long[] dst, int limit) throws IOException {
try {
GroupVIntUtil.readGroupVInts(..);
}
}
```
Finally, we remove the `readGroupVInts` method from subclasses, and make the base class `DataInput` implement as `final`(because nothing to optimize there anymore), the subclasses just override the `protected readGroupVInt()`. see [delta](https://github.com/apache/lucene/pull/12841#issuecomment-1851098019)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2401796608
I get confused that this PR was not merged yet it has a milestone set. Should the milestone be set to the other PR that has been merged instead (#13830) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Drop final modifier on the public DataInput.readGroupVInts method [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #13825:
URL: https://github.com/apache/lucene/pull/13825#issuecomment-2401859755
I've added migration and changes entry and cherry picked on branch_10_0, branch_10x and main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org