You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Marcus Falck <ma...@observer.se> on 2006/06/20 12:02:36 UTC
Modifying the stored norm type
Hi again,
After a lot of debugging and some API doc reading I have come to the
conclusion that the static encodeNorm method of the Similarity class
will encode my boost value into a single byte decimal number.
And I will loose a lot of resolution and will get severe rounding
errors.
(please correct me if I'm wrong)
Since I need the exact float value as boost representation this isn't
good enough in my case.
So I guess I will have to get lucene to store a 4 byte norm in the form
of a float instead of the single byte?
Is this do able or is it just madness? And will it slow the search
timings down or will it just eat more memory?
Again the alternative is spending 0.8 million $ on fast.
/
Regards
Marcus
Re: Modifying the stored norm type
Posted by Yonik Seeley <ys...@gmail.com>.
On 6/20/06, Marcus Falck <ma...@observer.se> wrote:
> So I guess I will have to get lucene to store a 4 byte norm in the form
> of a float instead of the single byte?
Can you store the float in a field instead? That seems like it might
be a bit easier than modifying how lucene stores norms.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-06-20 at 13:59 +0200, karl wettin wrote:
> On Tue, 2006-06-20 at 12:02 +0200, Marcus Falck wrote:
>
> > So I guess I will have to get lucene to store a 4 byte norm in the
> > form of a float instead of the single byte?
> >
> > Is this do able or is it just madness? And will it slow the search
> > timings down or will it just eat more memory?
>
> It is not to hard to refactor Lucene (back) in to doing that. I did it
> once. That code is no longer around, but it took something like an hour.
I would personally start in the SegmentWriter/Reader-end.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-06-20 at 12:02 +0200, Marcus Falck wrote:
> So I guess I will have to get lucene to store a 4 byte norm in the
> form of a float instead of the single byte?
>
> Is this do able or is it just madness? And will it slow the search
> timings down or will it just eat more memory?
It is not to hard to refactor Lucene (back) in to doing that. I did it
once. That code is no longer around, but it took something like an hour.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by Yonik Seeley <ys...@gmail.com>.
On 6/21/06, karl wettin <ka...@snigel.net> wrote:
> Marcus is trying to use the norms to enforce results in chronological
> order when matching a TB-sized corpus. He can't get any speed by sorting
> on a date field.
Once a FieldCache entry is populated, sorting on a DateField should be
about the same speed as sorting by score.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by Yonik Seeley <ys...@gmail.com>.
On 6/22/06, karl wettin <ka...@snigel.net> wrote:
> I tried to make a quick and dirty proof of concept, but noticed that no
> matter what order TermDocs return the documents, the collector get
> ascending document number order.
TermDocs should also always return documents in ascending order for a
single term (that's how they are stored internally).
> What dictates the collected document order?
The scorers, and a general agreement. The older BooleanScorer did
some document re-arrangement, but still relied on every other Scorer
delivered documents in-order.
The new BooleanScorer2 (default since 1.9) delivers docs in order.
Going in document order makes the whole process of matching/scoring
efficient.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by karl wettin <ka...@snigel.net>.
On Wed, 2006-06-21 at 19:32 +0200, Paul Elschot wrote:
>
> > TermDocs in reversed chronological order
>
> There is no need to write extra code for that, the documents would be
> collected oldest first, newest last.
I tried to make a quick and dirty proof of concept, but noticed that no
matter what order TermDocs return the documents, the collector get
ascending document number order.
Just as you write.
What dictates the collected document order?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by Paul Elschot <pa...@xs4all.nl>.
On Wednesday 21 June 2006 12:13, karl wettin wrote:
> On Tue, 2006-06-20 at 18:01 +0200, Paul Elschot wrote:
> > On Tuesday 20 June 2006 12:02, Marcus Falck wrote:
>
> > encodeNorm method of the Similarity class will encode my boost value
> > into a single byte decimal number. And I will loose a lot of
> > resolution and will get severe rounding errors.
>
> > Are 256 different values enough for your case?
>
> Marcus is trying to use the norms to enforce results in chronological
> order when matching a TB-sized corpus. He can't get any speed by sorting
> on a date field.
>
> Here is an idea:
>
> Never delete documents. Use unsafe document number as system clock. Make
Deleting documents does not change the order of the remaining ones.
> sure TermDocs always return references in reversed chronological order
There is no need to write extra code for that, the documents would be
collected oldest first, newest last.
> and write a HitCollector that does not re-order.
>
> That should work, right?
In case you need oldest first, yes.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-06-20 at 18:01 +0200, Paul Elschot wrote:
> On Tuesday 20 June 2006 12:02, Marcus Falck wrote:
> encodeNorm method of the Similarity class will encode my boost value
> into a single byte decimal number. And I will loose a lot of
> resolution and will get severe rounding errors.
> Are 256 different values enough for your case?
Marcus is trying to use the norms to enforce results in chronological
order when matching a TB-sized corpus. He can't get any speed by sorting
on a date field.
Here is an idea:
Never delete documents. Use unsafe document number as system clock. Make
sure TermDocs always return references in reversed chronological order
and write a HitCollector that does not re-order.
That should work, right?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Modifying the stored norm type
Posted by Paul Elschot <pa...@xs4all.nl>.
On Tuesday 20 June 2006 12:02, Marcus Falck wrote:
> Hi again,
>
>
>
> After a lot of debugging and some API doc reading I have come to the
> conclusion that the static encodeNorm method of the Similarity class
> will encode my boost value into a single byte decimal number.
>
> And I will loose a lot of resolution and will get severe rounding
> errors.
>
> (please correct me if I'm wrong)
>
>
>
> Since I need the exact float value as boost representation this isn't
> good enough in my case.
An exact float value is a bit of an oxymoron.
How exact do you need it to be?
The range of values that can be encoded by the existing encodeNorm()
and decodeNorm() is quite big (about 10e-10 to 10e+10 iirc),
and since there are only 255 possible values in there (excluding 0),
the rounding errors can be severe indeed.
However, with a smaller range, the rounding errors would also be smaller.
Are 256 different values enough for your case?
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org