You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stefan Gusenbauer <st...@kbse.net> on 2005/11/02 15:10:44 UTC

Vector Model and Relevance Feedback

I've some thoughts about Lucene and Relevance Feedback. I want to 
implement some variation of the Roccio Formula and there is the problem.
The formula is like this:

Query(new) = alpha * Query(old) + beta * Sum(Relevant Documents) - gamma 
* Sum(Non Relevant Documents)

The relevant documents in this formula should be in a vector 
representation. This is the problem If I work with TermFreqVectors then 
the vectors are not equally long and contains different terms. My 
solution now is to take the TermFreqVectors and minimize them to the 
least common multiple and perform then the computation.
So my questions are:
Is this the only way to do so? ( I hope so not)
Is there an add on for lucene to get a real vector representation?
Does anyone has experiences with this issue?
Thanks
Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Vector Model and Relevance Feedback

Posted by mark harwood <ma...@yahoo.co.uk>.
> since Lucene doesn't 
> currently support negative boosts 

See here for an approach to negative boosts:

http://wiki.apache.org/jakarta-lucene/CommunityContributions


Cheers
Mark




	
	
		
___________________________________________________________ 
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Vector Model and Relevance Feedback

Posted by Grant Ingersoll <gs...@syr.edu>.
Others can correct me if I am wrong, but I don't think a "pure" Rochio 
feedback loop is possible in the current state, since Lucene doesn't 
currently support negative boosts 
(http://lucene.apache.org/java/docs/queryparsersyntax.html).  Having 
said that, what we do, in a nutshell is similar to what you describe:
For the positive examples, store the terms and a boost factor.  The 
boost factor is the frequency of the term across all the positive 
examples multiplied by beta.
Then for the negative examples, decrement the boost factor by gamma 
times the frequency of the term in all the negative examples.  Remove 
any terms that have a boost of zero or less.

In the end, you construct a new query out of the terms and boosts that 
you can submit.  I think it is more of an approximation of Rochio, but 
have had good results from it.  You also probably want to limit the 
number of terms per document you add, at least if you are concerned 
about performance.
    
-Grant

Stefan Gusenbauer wrote:

> I've some thoughts about Lucene and Relevance Feedback. I want to 
> implement some variation of the Roccio Formula and there is the problem.
> The formula is like this:
>
> Query(new) = alpha * Query(old) + beta * Sum(Relevant Documents) - 
> gamma * Sum(Non Relevant Documents)
>
> The relevant documents in this formula should be in a vector 
> representation. This is the problem If I work with TermFreqVectors 
> then the vectors are not equally long and contains different terms. My 
> solution now is to take the TermFreqVectors and minimize them to the 
> least common multiple and perform then the computation.
> So my questions are:
> Is this the only way to do so? ( I hope so not)
> Is there an add on for lucene to get a real vector representation?
> Does anyone has experiences with this issue?
> Thanks
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-- 
------------------------------------------------------------------- 
Grant Ingersoll 
Sr. Software Engineer 
Center for Natural Language Processing 
Syracuse University 
School of Information Studies 
337 Hinds Hall 
Syracuse, NY 13244 

http://www.cnlp.org 
Voice:  315-443-5484 
Fax: 315-443-6886 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Vector Model and Relevance Feedback

Posted by Ian Soboroff <ia...@nist.gov>.
Stefan Gusenbauer <st...@kbse.net> writes:

> Is there an add on for lucene to get a real vector representation?
> Does anyone has experiences with this issue?

No code, but some small thinking.  You can do hacks with boosts and
whatnot, but I think in the end you really want a new Query subclass
(and the other machinery to make that work).

I've seen some of the boost hacks, and, ok, Rocchio is something of a
hack (at least the way it's usually done), but adding a hack to a hack
is just too much.  Next, some joker is going to come along and want to
to probabilistic LSI or language models and then you're up the creek
;-)

I have a code infrastructure to handle nearly any VSM-style retrieval
function (including Okapi BM25), but it's waiting for the low-level
VSM query structure to make it worthwhile.  If someone starts work in
this area, I'd love to help out to the extent I can.

Ian



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org