You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by EDMOND KEMOKAI <ek...@gmail.com> on 2007/01/22 19:33:31 UTC
Lucene Internals question
Hi All
This is a question for those familiar with lucene document scoring. How does
it compare with googles PageRank or HITS, or are they very different? I have
being looking at the PageRank algorithm but I'll need to brush-off my math
skills before delving into it:)
--
"talk trash and carry a small stick."
PAUL KRUGMAN (NYT)
Re: Lucene Internals question
Posted by Yonik Seeley <yo...@apache.org>.
On 1/23/07, Grant Ingersoll <gr...@gmail.com> wrote:
> You might also be interested in https://issues.apache.org/jira/browse/
> LUCENE-755 (aka the Payloads patch) which will enable storing
> information at the token level and allow for plugging in more scoring
> options related to it.
>
> There has been a variety of discussions over on java-dev related to
> topics that would enable things like PageRank, etc. at a lower level
> in Lucene. Search java-dev for "Flexible Indexing", "Payloads", etc.
If page-rank itself is static/per-document, one could already handle
that case with lucene today.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Internals question
Posted by Grant Ingersoll <gr...@gmail.com>.
You might also be interested in https://issues.apache.org/jira/browse/
LUCENE-755 (aka the Payloads patch) which will enable storing
information at the token level and allow for plugging in more scoring
options related to it.
There has been a variety of discussions over on java-dev related to
topics that would enable things like PageRank, etc. at a lower level
in Lucene. Search java-dev for "Flexible Indexing", "Payloads", etc.
You will also find additional Lucene scoring information at http://
lucene.apache.org/java/docs/scoring.html
-Grant
On Jan 22, 2007, at 4:44 PM, Nicolas Lalevée wrote:
> Le Lundi 22 Janvier 2007 20:44, EDMOND KEMOKAI a écrit :
>> Hmm..doesn't lucene scoring determine how relevant a document is
>> to your
>> query? That is what PageRank and HITS do as well, I believe. Page and
>> document are the same, if you want to index a page you'll
>> obviously try to
>> convert it into a document. PageRank does link analysis to
>> determine how
>> relevant that page is as it relates to the query you entered, does
>> lucene
>> have something similar? How does lucene determine between two
>> documents
>> which one should score higher if they both contain a certain term?
>> Google
>> uses PageRank to make that determination, how does lucene do it?
>
> The thing is that Google can use both its PageRank and Lucene.
>
> The PageRank caculation doesn't depend on the query. It's value is
> quite
> absolute. It depends of how the other pages refer to it.
>
> And the Lucene scoring is calculating how a document (document is
> the generic
> Lucene term, beacause Lucene can be used for other things that web
> search)
> match the query. Searching for "foo bar", a document with no "foo",
> no "bar"
> in it it scored at 0, even more not scored at all. A document with
> one "foo",
> but no "bar" will be scored to 0.5, with one "foo" and one "bar",
> separated
> with 15 words, will be scored 0.8, and a document with "foo bar",
> scored to
> 1.0.
>
> Then you can combine the both, as Google is probably doing. Same
> exemple :
> - a page with no "foo", neither "bar", whatever is its page rank,
> won't be
> scored.
> - a page with a "foo" and a "bar", with a low page rank, let say 0.5
> - a page with a "foo" and a "bar", with a high page rank, let say 0.9
>
> If you want to know more about Lucene scoring and Page rank, you
> should go see
> Nutch and its mailing list : http://lucene.apache.org/nutch/
>
> Nicolas
>
>>
>> On 1/22/07, Nicolas Lalevée <ni...@anyware-tech.com> wrote:
>>> Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit:
>>>> Hi All
>>>> This is a question for those familiar with lucene document
>>>> scoring. How
>>>> does it compare with googles PageRank or HITS, or are they very
>>>
>>> different?
>>>
>>>> I have being looking at the PageRank algorithm but I'll need to
>>>
>>> brush-off
>>>
>>>> my math skills before delving into it:)
>>>
>>> In fact Lucene is just a search engine. Then you can use the search
>>> engine to
>>> search in web pages, like Nutch is using Lucene. And Google is
>>> more like
>>> Nutch : a web crawler plus a web-search engine. So when you are
>>> taking
>>> about
>>> page raking, it has nothing to do with Lucene scoring. Lucene
>>> scoring is
>>> how
>>> about the result entry match your query. Page raking is more
>>> about how
>>> relevant is the web page. So for a document, the Lucene scoring
>>> depends
>>> on the query, and the page raking is quite absolute.
>>>
>>> Nicolas
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
------------------------------------------------------
Grant Ingersoll
http://www.grantingersoll.com/
http://www.paperoftheweek.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Internals question
Posted by Nicolas Lalevée <ni...@anyware-tech.com>.
Le Lundi 22 Janvier 2007 20:44, EDMOND KEMOKAI a écrit :
> Hmm..doesn't lucene scoring determine how relevant a document is to your
> query? That is what PageRank and HITS do as well, I believe. Page and
> document are the same, if you want to index a page you'll obviously try to
> convert it into a document. PageRank does link analysis to determine how
> relevant that page is as it relates to the query you entered, does lucene
> have something similar? How does lucene determine between two documents
> which one should score higher if they both contain a certain term? Google
> uses PageRank to make that determination, how does lucene do it?
The thing is that Google can use both its PageRank and Lucene.
The PageRank caculation doesn't depend on the query. It's value is quite
absolute. It depends of how the other pages refer to it.
And the Lucene scoring is calculating how a document (document is the generic
Lucene term, beacause Lucene can be used for other things that web search)
match the query. Searching for "foo bar", a document with no "foo", no "bar"
in it it scored at 0, even more not scored at all. A document with one "foo",
but no "bar" will be scored to 0.5, with one "foo" and one "bar", separated
with 15 words, will be scored 0.8, and a document with "foo bar", scored to
1.0.
Then you can combine the both, as Google is probably doing. Same exemple :
- a page with no "foo", neither "bar", whatever is its page rank, won't be
scored.
- a page with a "foo" and a "bar", with a low page rank, let say 0.5
- a page with a "foo" and a "bar", with a high page rank, let say 0.9
If you want to know more about Lucene scoring and Page rank, you should go see
Nutch and its mailing list : http://lucene.apache.org/nutch/
Nicolas
>
> On 1/22/07, Nicolas Lalevée <ni...@anyware-tech.com> wrote:
> > Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit:
> > > Hi All
> > > This is a question for those familiar with lucene document scoring. How
> > > does it compare with googles PageRank or HITS, or are they very
> >
> > different?
> >
> > > I have being looking at the PageRank algorithm but I'll need to
> >
> > brush-off
> >
> > > my math skills before delving into it:)
> >
> > In fact Lucene is just a search engine. Then you can use the search
> > engine to
> > search in web pages, like Nutch is using Lucene. And Google is more like
> > Nutch : a web crawler plus a web-search engine. So when you are taking
> > about
> > page raking, it has nothing to do with Lucene scoring. Lucene scoring is
> > how
> > about the result entry match your query. Page raking is more about how
> > relevant is the web page. So for a document, the Lucene scoring depends
> > on the query, and the page raking is quite absolute.
> >
> > Nicolas
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Internals question
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hello,
Lucene gives the best documents for a given query and PageRank uses
citation
analysis with similar results but requires a large set of metadata to
complete.
Scoring in Lucene delivers pure search while PageRank attempts to
establish source authority.
I''m not strong in math, those who are can find an explanation of the
latter here:
http://www.ams.org/featurecolumn/archive/pagerank.html
Regards,
Peter W.
On Jan 22, 2007, at 12:00 PM, Mark Miller wrote:
> Well first Lucene checks all of the other documents in the world
> for any that that refer to the document that your adding to
> Lucene...and then...oh wait...
>
> http://lucene.apache.org/java/docs/api/org/apache/lucene/search/
> Similarity.html
>
> EDMOND KEMOKAI wrote:
>> Hmm..doesn't lucene scoring determine how relevant a document is
>> to your
>> query? That is what PageRank and HITS do as well, I believe. Page and
>> document are the same, if you want to index a page you'll
>> obviously try to
>> convert it into a document. PageRank does link analysis to
>> determine how
>> relevant that page is as it relates to the query you entered, does
>> lucene
>> have something similar? How does lucene determine between two
>> documents
>> which one should score higher if they both contain a certain term?
>> Google
>> uses PageRank to make that determination, how does lucene do it?
>>
>> On 1/22/07, Nicolas Lalevée <ni...@anyware-tech.com> wrote:
>>>
>>> Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit:
>>> > Hi All
>>> > This is a question for those familiar with lucene document
>>> scoring. How
>>> > does it compare with googles PageRank or HITS, or are they very
>>> different?
>>> > I have being looking at the PageRank algorithm but I'll need to
>>> brush-off
>>> > my math skills before delving into it:)
>>>
>>> In fact Lucene is just a search engine. Then you can use the
>>> search engine
>>> to
>>> search in web pages, like Nutch is using Lucene. And Google is
>>> more like
>>> Nutch : a web crawler plus a web-search engine. So when you are
>>> taking
>>> about
>>> page raking, it has nothing to do with Lucene scoring. Lucene
>>> scoring is
>>> how
>>> about the result entry match your query. Page raking is more
>>> about how
>>> relevant is the web page. So for a document, the Lucene scoring
>>> depends on
>>> the query, and the page raking is quite absolute.
>>>
>>> Nicolas
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Internals question
Posted by Mark Miller <ma...@gmail.com>.
Well first Lucene checks all of the other documents in the world for any
that that refer to the document that your adding to Lucene...and
then...oh wait...
http://lucene.apache.org/java/docs/api/org/apache/lucene/search/Similarity.html
EDMOND KEMOKAI wrote:
> Hmm..doesn't lucene scoring determine how relevant a document is to your
> query? That is what PageRank and HITS do as well, I believe. Page and
> document are the same, if you want to index a page you'll obviously
> try to
> convert it into a document. PageRank does link analysis to determine how
> relevant that page is as it relates to the query you entered, does lucene
> have something similar? How does lucene determine between two documents
> which one should score higher if they both contain a certain term? Google
> uses PageRank to make that determination, how does lucene do it?
>
> On 1/22/07, Nicolas Lalevée <ni...@anyware-tech.com> wrote:
>>
>> Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit:
>> > Hi All
>> > This is a question for those familiar with lucene document scoring.
>> How
>> > does it compare with googles PageRank or HITS, or are they very
>> different?
>> > I have being looking at the PageRank algorithm but I'll need to
>> brush-off
>> > my math skills before delving into it:)
>>
>> In fact Lucene is just a search engine. Then you can use the search
>> engine
>> to
>> search in web pages, like Nutch is using Lucene. And Google is more like
>> Nutch : a web crawler plus a web-search engine. So when you are taking
>> about
>> page raking, it has nothing to do with Lucene scoring. Lucene scoring is
>> how
>> about the result entry match your query. Page raking is more about how
>> relevant is the web page. So for a document, the Lucene scoring
>> depends on
>> the query, and the page raking is quite absolute.
>>
>> Nicolas
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Internals question
Posted by EDMOND KEMOKAI <ek...@gmail.com>.
Hmm..doesn't lucene scoring determine how relevant a document is to your
query? That is what PageRank and HITS do as well, I believe. Page and
document are the same, if you want to index a page you'll obviously try to
convert it into a document. PageRank does link analysis to determine how
relevant that page is as it relates to the query you entered, does lucene
have something similar? How does lucene determine between two documents
which one should score higher if they both contain a certain term? Google
uses PageRank to make that determination, how does lucene do it?
On 1/22/07, Nicolas Lalevée <ni...@anyware-tech.com> wrote:
>
> Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit:
> > Hi All
> > This is a question for those familiar with lucene document scoring. How
> > does it compare with googles PageRank or HITS, or are they very
> different?
> > I have being looking at the PageRank algorithm but I'll need to
> brush-off
> > my math skills before delving into it:)
>
> In fact Lucene is just a search engine. Then you can use the search engine
> to
> search in web pages, like Nutch is using Lucene. And Google is more like
> Nutch : a web crawler plus a web-search engine. So when you are taking
> about
> page raking, it has nothing to do with Lucene scoring. Lucene scoring is
> how
> about the result entry match your query. Page raking is more about how
> relevant is the web page. So for a document, the Lucene scoring depends on
> the query, and the page raking is quite absolute.
>
> Nicolas
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
"talk trash and carry a small stick."
PAUL KRUGMAN (NYT)
Re: Lucene Internals question
Posted by Nicolas Lalevée <ni...@anyware-tech.com>.
Le Lundi 22 Janvier 2007 19:33, EDMOND KEMOKAI a écrit :
> Hi All
> This is a question for those familiar with lucene document scoring. How
> does it compare with googles PageRank or HITS, or are they very different?
> I have being looking at the PageRank algorithm but I'll need to brush-off
> my math skills before delving into it:)
In fact Lucene is just a search engine. Then you can use the search engine to
search in web pages, like Nutch is using Lucene. And Google is more like
Nutch : a web crawler plus a web-search engine. So when you are taking about
page raking, it has nothing to do with Lucene scoring. Lucene scoring is how
about the result entry match your query. Page raking is more about how
relevant is the web page. So for a document, the Lucene scoring depends on
the query, and the page raking is quite absolute.
Nicolas
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org