You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Marc Davenport <ma...@cargurus.com.INVALID> on 2024/08/05 16:07:27 UTC
Converting docid to uid
Hello,
Right now our implementation retrieves our UID for our records from the
topdocs by calling IndexSearcher.doc(docid, fieldToLoad) (Deprecated) with
the UID as the only field. I'm looking to replace this with the
appropriate call to IndexSearcher.storedFields(). This feels a little
inefficient since it could be visiting each segment on disk. Some of our
searches are for an abusive 10k results (I would love to change this, but
that's the system as it exists). If I only want a single field, is there
a better way to retrieve these? Should I be retrieving the numeric docvals
and advancing the iterator? I've been looking around the internet for
different strategies and I'm kind of muddled with older blog posts vs the
current state of the codebase. Can I cache the relationship between the
docId and the UID as long as I clear it whenever I commit?
Thank you,
Marc
Re: Converting docid to uid
Posted by Marc Davenport <ma...@cargurus.com.INVALID>.
Hello,
To close the circle on this. I found that using the docvalues for this was
on the order of 10x faster for our purposes.
Snippet that someone might find useful in the future:
LeafReaderContext context =
indexReader.leaves().get(ReaderUtil.subIndex(docId, indexReader.leaves()));
NumericDocValues uidDocValues =
context.reader().getNumericDocValues(LISTING_ID_FIELDNAME);
if (uidDocValues != null && uidDocValues.advanceExact(docId -
context.docBase)) {
return uidDocValues.longValue();
}
// not found, handle error
Previous approach using stored field for comparison:
Set<String> listingIdField = ImmutableSet.of(LISTING_ID_FIELDNAME);
Document d = doc(docId, listingIdField);
IndexableField value = d.getField(LISTING_ID_FIELDNAME);
if (value != null && value.numericValue() != null) {
return value.numericValue().longValue();
}
// not found, handle error
I ran the two in parallel for a while to make sure that the resulting value
matched and never had one miss.
We did end up caching the relationship, but the cache is bundled with the
wrapper to the index reader which is switched out at every update/commit.
This has been stable for us. The cache went in before I was able to switch
retrieving through the numericDocValues. We probably wouldn't have added
the cache if we had already made the switch.
Thank Michael et al!
Marc
On Tue, Aug 6, 2024 at 5:07 PM Michael Sokolov <ms...@gmail.com> wrote:
> You could switch to DocValues, and it would probably be more efficient
> if you are only retrieving a single stored field but you have a lot of
> other ones in the index since stored fields are stored together and
> have to be decoded together. As far as visiting every segment on disk
> I'm not sure what you mean -- if you mean Lucene segments this isn't
> really germane since both DocValues and StoredFields are stored in
> segments and the segment visitation pattern is going to be the same
> for both. Finally re: caching I wouldn't recommend it but the
> approach you describe could be made to work. To be efficient about it
> you'd want to have a per-segment cache, but this gets into poking
> around in Lucene internals - you might as well just use its data
> structures; they should be pretty efficient and a cache wouldn't
> likely win you very much and just lead to trouble,
>
> On Mon, Aug 5, 2024 at 12:08 PM Marc Davenport
> <ma...@cargurus.com.invalid> wrote:
> >
> > Hello,
> > Right now our implementation retrieves our UID for our records from the
> > topdocs by calling IndexSearcher.doc(docid, fieldToLoad) (Deprecated)
> with
> > the UID as the only field. I'm looking to replace this with the
> > appropriate call to IndexSearcher.storedFields(). This feels a little
> > inefficient since it could be visiting each segment on disk. Some of our
> > searches are for an abusive 10k results (I would love to change this, but
> > that's the system as it exists). If I only want a single field, is
> there
> > a better way to retrieve these? Should I be retrieving the numeric
> docvals
> > and advancing the iterator? I've been looking around the internet for
> > different strategies and I'm kind of muddled with older blog posts vs the
> > current state of the codebase. Can I cache the relationship between the
> > docId and the UID as long as I clear it whenever I commit?
> > Thank you,
> > Marc
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Converting docid to uid
Posted by Michael Sokolov <ms...@gmail.com>.
You could switch to DocValues, and it would probably be more efficient
if you are only retrieving a single stored field but you have a lot of
other ones in the index since stored fields are stored together and
have to be decoded together. As far as visiting every segment on disk
I'm not sure what you mean -- if you mean Lucene segments this isn't
really germane since both DocValues and StoredFields are stored in
segments and the segment visitation pattern is going to be the same
for both. Finally re: caching I wouldn't recommend it but the
approach you describe could be made to work. To be efficient about it
you'd want to have a per-segment cache, but this gets into poking
around in Lucene internals - you might as well just use its data
structures; they should be pretty efficient and a cache wouldn't
likely win you very much and just lead to trouble,
On Mon, Aug 5, 2024 at 12:08 PM Marc Davenport
<ma...@cargurus.com.invalid> wrote:
>
> Hello,
> Right now our implementation retrieves our UID for our records from the
> topdocs by calling IndexSearcher.doc(docid, fieldToLoad) (Deprecated) with
> the UID as the only field. I'm looking to replace this with the
> appropriate call to IndexSearcher.storedFields(). This feels a little
> inefficient since it could be visiting each segment on disk. Some of our
> searches are for an abusive 10k results (I would love to change this, but
> that's the system as it exists). If I only want a single field, is there
> a better way to retrieve these? Should I be retrieving the numeric docvals
> and advancing the iterator? I've been looking around the internet for
> different strategies and I'm kind of muddled with older blog posts vs the
> current state of the codebase. Can I cache the relationship between the
> docId and the UID as long as I clear it whenever I commit?
> Thank you,
> Marc
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org