You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Sharon W Tam <st...@mit.edu> on 2013/04/02 16:10:01 UTC
Re: Indexing Term Frequency Vectors
Thanks for your help, Adrien. But unfortunately, my term frequencies will
be partial counts so they won't be integers, And finding a common
denominator and scaling the rest of the frequencies accordingly will affect
the relative lengths of the documents which will affect the Lucene scoring
because the length of the documents is taken into account in the scoring.
Are there any other ideas?
On Thu, Mar 28, 2013 at 9:06 PM, Adrien Grand <jp...@gmail.com> wrote:
> Hi,
>
> On Thu, Mar 28, 2013 at 8:25 PM, Sharon Tam <sh...@gmail.com> wrote:
> > I believe that when Lucene indexes documents, it generates counts for a
> > term by counting how many times the term appears in a particular
> document.
> > Instead of having Lucene do the counting, I want to do my own counting
> and
> > feed a term-frequency vector representation of a document directly into
> the
> > indexer which will take my counts and proceed to do the other processing
> > such as generating inverse document frequency. My term-frequencies may
> not
> > all be integers. Is there a way to do this?
>
> You could provide the indexer with arbitrary frequencies by creating a
> handcrafted TokenStream that repeats terms ${termFreq} times, but
> unfortunately, frequencies need to be strictly positive (> 0)
> integers.
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Indexing Term Frequency Vectors
Posted by Adrien Grand <jp...@gmail.com>.
Hi,
On Tue, Apr 9, 2013 at 5:24 PM, Sharon Tam <sh...@gmail.com> wrote:
> I tried following following this payloads tutorial to attach the term
> frequencies as payloads:
> http://searchhub.org/2009/08/05/getting-started-with-payloads/
>
> But I'm confused as to where I need to override the term frequency counter
> so that it uses my term frequencies. I think term frequency counts are
> calculated during indexing, so I don't think I can just write my own
> Similarity class?
This is correct, frequencies are computed at indexing time. I just
wanted to mention that you can influence scores based on payloads:
http://lucene.apache.org/core/4_2_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html#scorePayload(int,
int, int, org.apache.lucene.util.BytesRef)
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing Term Frequency Vectors
Posted by Adrien Grand <jp...@gmail.com>.
On Tue, Apr 2, 2013 at 4:10 PM, Sharon W Tam <st...@mit.edu> wrote:
> Are there any other ideas?
Since scoring seems to be what you are interested in, you could have a
look to payloads: there can store arbitrary data and can be used to
score matches.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org