You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Peter V. Metalnikov" <pe...@ssd-lab.com> on 2005/04/08 19:26:33 UTC
strange results given by boolean queries
Please, advice if I am doing something wrong or such a behaviour is normal:
when searching for two phrases using OR nutch returns the results as if the
phrases were united by AND, i.e. does not search for each phrase separately
but just looks for documents where both phrases are present.
--
With respect, Peter V. Metalnikov
Re: Incremental Crawling
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
Hi,
I would like to perform an incremental crawling using Nutch. What I want
to do is to configure Nutch in such a way that it should check for
expired pages and issue new crawls to the expires pages only.
Other requirements are:
Ability to inject new urls to the crawl database. When incremental
crawling begins, nutch should crawl the newly injected urls.
After an incremental crawl is completed, either a new search index
should be created or the previous search index should be updated.
Can anyone suggest how to achieve this?
Thanks,
Kannan
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com
Re: Incremental Crawling
Posted by Andrzej Bialecki <ab...@getopt.org>.
Doug Cutting wrote:
> Jérôme Charron wrote:
>
>> Then, the question is where the TheDateToCheck value comes from?
>> 1. From the previously indexed document (I know that this information
>> is stored): It certainly consumes more process time that the second
>> solution. My knowledge of Nutch internal is not enougth to know how to
>> retrieve quickly this information from the document's url... someone
>> can help us on this point?
>
>
> The most efficient place to store this would be in the pagedb. What's
> stored there currently is the nextFetch date and the fetchInterval. This
> could be changed to lastModified and fetchInterval, with nextFetch
> calculated as lastModified+fetchInterval. In UpdateDatabaseTool.java
> these can both be updated. If the lastModified has not changed then
> then fetchInterval can be increased accordingly.
If you remember, we discussed this some time ago. We came to a
conclusion that in order to properly support varying fetchInterval its
type needs to be changed to a float (or a range-reduced float like in
Lucene), and a lot of changes need to be done in the API between the
fetcher and plugins.
It would be a good time to start this now, when there are upcoming
incompatible changes in the API and disk formats... Within a couple days
I'll try to refresh the patchset I had then and put it up for review.
--
Best regards,
Andrzej Bialecki
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Incremental Crawling
Posted by Doug Cutting <cu...@nutch.org>.
Jérôme Charron wrote:
> Then, the question is where the TheDateToCheck value comes from?
> 1. From the previously indexed document (I know that this information is
> stored): It certainly consumes more process time that the second solution.
> My knowledge of Nutch internal is not enougth to know how to retrieve
> quickly this information from the document's url... someone can help us on
> this point?
The most efficient place to store this would be in the pagedb. What's
stored there currently is the nextFetch date and the fetchInterval.
This could be changed to lastModified and fetchInterval, with nextFetch
calculated as lastModified+fetchInterval. In UpdateDatabaseTool.java
these can both be updated. If the lastModified has not changed then
then fetchInterval can be increased accordingly.
Doug
Re: Incremental Crawling
Posted by Jérôme Charron <je...@gmail.com>.
> Can you please suggest how to go about implementing this? I would like
> to add this check.
In the HttpResponse class, just add something like (it uses the
If-Modified-Since header, not the HEAD method) :
reqStr.append("If-Modified-Since: ");
reqStr.append(TheDateToCheck);
reqStr.append("\r\n");
just before the last line:
reqStr.append("\r\n");
Then, you must correctly format the TheDateToCheck variable content (see
http://www.zvon.org/tmRFC/RFC2068/Output/chapter14.html#sub24 for a
description of If-Modified-Since header specifications).
Then, the question is where the TheDateToCheck value comes from?
1. From the previously indexed document (I know that this information is
stored): It certainly consumes more process time that the second solution.
My knowledge of Nutch internal is not enougth to know how to retrieve
quickly this information from the document's url... someone can help us on
this point?
2. From the last time you launch the fetching : So it is a global date that
you store in a file and retrieves in the HttpResponse class to perform the
date check...
You must change the Http.java class too, in order to correctly handle the
304 response code:
After the lines
} else if (code == 410) { // page is gone
throw new ResourceGone(url, "Http: " + code);
Just add something like:
} else if (code == 304) {
throw new HttpException("No modified: " + urlString);
(I hope that throwing the exception will not remove the previous one. I
think it is not the case, it is the ResourceGone exception that is uses to
indicate that a resource no more exists, but if someone can confirm....)
Jerome
--
http://motrech.free.fr/
http://frutch.free.fr/
Re: Incremental Crawling
Posted by Jérôme Charron <je...@gmail.com>.
> Can you please suggest how to go about implementing this? I would like
> to add this check.
In the HttpResponse class, just add something like (it uses the
If-Modified-Since header, not the HEAD method) :
reqStr.append("If-Modified-Since: ");
reqStr.append(TheDateToCheck);
reqStr.append("\r\n");
just before the last line:
reqStr.append("\r\n");
Then, you must correctly format the TheDateToCheck variable content (see
http://www.zvon.org/tmRFC/RFC2068/Output/chapter14.html#sub24 for a
description of If-Modified-Since header specifications).
Then, the question is where the TheDateToCheck value comes from?
1. From the previously indexed document (I know that this information is
stored): It certainly consumes more process time that the second solution.
My knowledge of Nutch internal is not enougth to know how to retrieve
quickly this information from the document's url... someone can help us on
this point?
2. From the last time you launch the fetching : So it is a global date that
you store in a file and retrieves in the HttpResponse class to perform the
date check...
You must change the Http.java class too, in order to correctly handle the
304 response code:
After the lines
} else if (code == 410) { // page is gone
throw new ResourceGone(url, "Http: " + code);
Just add something like:
} else if (code == 304) {
throw new HttpException("No modified: " + urlString);
(I hope that throwing the exception will not remove the previous one. I
think it is not the case, it is the ResourceGone exception that is uses to
indicate that a resource no more exists, but if someone can confirm....)
Jerome
--
http://motrech.free.fr/
http://frutch.free.fr/
Re: Incremental Crawling
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
Can you please suggest how to go about implementing this? I would like
to add this check.
On Thu, 2005-04-21 at 13:14, Jérôme Charron wrote:
> >
> > What i mean by expired pages is those pages whose last Modified date has
> > changed since last fetch.
> > Whole-web crawling fetches all pages that are due to be fetched (e.g,
> > every 30 days). These pages may not have actually changed in content. I
> > would like to know if there is any way to tell Nutch to compare the last
> > modified date and fetch the page only if the date is different from what
> > is there in the index. I think this way we can save time by fetching and
> > indexing only the modified pages while re-crawling the same site after
> > some time.
>
>
> I have suggested many time ago use the HEAD method or the GET header
> If-Modified-Since (as sugested by Otis) in order to fetch only changed
> documents.
> The discussion is here:
> http://www.mail-archive.com/nutch-dev@incubator.apache.org/msg00091.html
> But actually I don't find time to implement this feature...
>
> Jerome
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com
Re: Incremental Crawling
Posted by Jérôme Charron <je...@gmail.com>.
>
> What i mean by expired pages is those pages whose last Modified date has
> changed since last fetch.
> Whole-web crawling fetches all pages that are due to be fetched (e.g,
> every 30 days). These pages may not have actually changed in content. I
> would like to know if there is any way to tell Nutch to compare the last
> modified date and fetch the page only if the date is different from what
> is there in the index. I think this way we can save time by fetching and
> indexing only the modified pages while re-crawling the same site after
> some time.
I have suggested many time ago use the HEAD method or the GET header
If-Modified-Since (as sugested by Otis) in order to fetch only changed
documents.
The discussion is here:
http://www.mail-archive.com/nutch-dev@incubator.apache.org/msg00091.html
But actually I don't find time to implement this feature...
Jerome
--
http://motrech.free.fr/
http://frutch.free.fr/
Re: Incremental Crawling
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
What i mean by expired pages is those pages whose last Modified date has
changed since last fetch.
Whole-web crawling fetches all pages that are due to be fetched (e.g,
every 30 days). These pages may not have actually changed in content. I
would like to know if there is any way to tell Nutch to compare the last
modified date and fetch the page only if the date is different from what
is there in the index. I think this way we can save time by fetching and
indexing only the modified pages while re-crawling the same site after
some time.
Kannan
On Tue, 2005-04-19 at 21:38, Doug Cutting wrote:
> Kannan Sundaramoorthy wrote:
> > I would like to perform an incremental crawling using Nutch. What I want
> > to do is to configure Nutch in such a way that it should check for
> > expired pages and issue new crawls to the expires pages only.
> > Other requirements are:
> > 1. Ability to inject new urls to the crawl database. When
> > incremental crawling begins, nutch should crawl the newly
> > injected urls.
> >
> > 2. After an incremental crawl is completed, either a new search
> > index should be created or the previous search index should be
> > updated.
> >
> > Can anyone suggest how to achieve this?
>
> This sounds like the "Whole-web Crawling" as described in the tutorial:
>
> http://incubator.apache.org/nutch/tutorial.html#Whole-web+Crawling
>
> By default this method will expire and recrawl urls every 30 days.
>
> Doug
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com
Re: Incremental Crawling
Posted by Doug Cutting <cu...@nutch.org>.
Kannan Sundaramoorthy wrote:
> I would like to perform an incremental crawling using Nutch. What I want
> to do is to configure Nutch in such a way that it should check for
> expired pages and issue new crawls to the expires pages only.
> Other requirements are:
> 1. Ability to inject new urls to the crawl database. When
> incremental crawling begins, nutch should crawl the newly
> injected urls.
>
> 2. After an incremental crawl is completed, either a new search
> index should be created or the previous search index should be
> updated.
>
> Can anyone suggest how to achieve this?
This sounds like the "Whole-web Crawling" as described in the tutorial:
http://incubator.apache.org/nutch/tutorial.html#Whole-web+Crawling
By default this method will expire and recrawl urls every 30 days.
Doug
Incremental Crawling
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
Hi,
I would like to perform an incremental crawling using Nutch. What I want
to do is to configure Nutch in such a way that it should check for
expired pages and issue new crawls to the expires pages only.
Other requirements are:
1. Ability to inject new urls to the crawl database. When
incremental crawling begins, nutch should crawl the newly
injected urls.
2. After an incremental crawl is completed, either a new search
index should be created or the previous search index should be
updated.
Can anyone suggest how to achieve this?
Thanks,
Kannan
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com
Re: Nutch scoring algorithm
Posted by Andy Liu <an...@gmail.com>.
You're correct, the fieldNorm is calculated at index time.
The URL and anchor boosts are query time boosts that can be changed
within the query-basic plugin. There's been talk about ripping these
out and placing them in the conf file, but I'm not sure if it's been
done yet.
On Apr 12, 2005 1:50 AM, Kannan Sundaramoorthy
<ka...@cognizant.com> wrote:
>
> Hi,
> Thanks for the explanation. I need some more info. I understand that
> fieldNorm is byte-encoded normalization factor for the named field of
> every document. This value is returned by norms(String field) of
> SegmentReader class. Is this normalization factor calculated at index
> time itself and is just read during searching?
>
> I used explain to see boosts for different fields. Please see the
> details below. As I see from explanation, "url" field is assigned a
> boost of 4.0 and "anchor" field is assigned a boost of 2.0. Please
> suggest me how I can alter boost values for different fields. Does it
> need any configuration change during indexing itself?
>
> page
>
> * docNo = 1
> * segment = 20050411183746
> * digest = 3835653251e4598bee61618b1c64804c
> * boost = 1.8572323
> * lastModified = 1113224620000
> * contentLength = 347
> * primaryType = text
> * subType = html
> * url = http://localhost:8080/none.html
> * title = None Document
>
> score for query: none
>
> * 1.5042199 = sum of:
> o 0.4181689 = weight(url:none^4.0 in 1), product of:
> + 0.8728715 = queryWeight(url:none^4.0), product of:
> # 4.0 = boost
> # 1.9162908 = idf(docFreq=1)
> # 0.11387514 = queryNorm
> + 0.4790727 = fieldWeight(url:none in 1), product of:
> # 1.0 = tf(termFreq(url:none)=1)
> # 1.9162908 = idf(docFreq=1)
> # 0.25 = fieldNorm(field=url, doc=1)
> o 1.0349152 = weight(anchor:none^2.0 in 1), product of:
> + 0.43643576 = queryWeight(anchor:none^2.0), product of:
> # 2.0 = boost
> # 1.9162908 = idf(docFreq=1)
> # 0.11387514 = queryNorm
> + 2.3712888 = fieldWeight(anchor:none in 1), product of:
> # 1.4142135 = tf(termFreq(anchor:none)=2)
> # 1.9162908 = idf(docFreq=1)
> # 0.875 = fieldNorm(field=anchor, doc=1)
> o 0.05113577 = weight(content:none in 1), product of:
> + 0.21821788 = queryWeight(content:none), product of:
> # 1.9162908 = idf(docFreq=1)
> # 0.11387514 = queryNorm
> + 0.23433356 = fieldWeight(content:none in 1), product of:
> # 2.236068 = tf(termFreq(content:none)=5)
> # 1.9162908 = idf(docFreq=1)
> # 0.0546875 = fieldNorm(field=content, doc=1)
>
> Thanks,
> Kannan
> On Mon, 2005-04-11 at 17:47, Andy Liu wrote:
> > fieldNorm is lengthNorm * document boost. The lengthNorm formula is
> > defined within Lucene's similarity class (which is a function of the
> > number of terms within the document) and the document boost is
> > calculated in IndexSegment.java .
> >
> > Nutch assigns different boosts to each field so that you can tune your
> > search results. For example, you can use explain to see if anchor
> > matches are too strong, and adjust accordingly.
> >
> > Andy
> >
> > On Apr 11, 2005 12:17 AM, Kannan Sundaramoorthy
> > <ka...@cognizant.com> wrote:
> > >
> > > Hi,
> > > I am trying to understand how Nutch computes score for each document. I
> > > could figure out how tf, idf and queryNorm are computed but I do not
> > > understand how fieldNorm (normalisation for each field) value is
> > > computed. It seems to be a magic number for me and this is where Nutch
> > > seems to differ from Lucene in computing score.
> > > Also Nutch assigns different boosts for different fields (e.g, 4.0 for
> > > url field) and uses this value while computing queryWeight. Can anyone
> > > explain these please?
> > >
> > > Thanks,
> > > Kannan
> > >
> > > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> > > If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> > > Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> > > prohibited and may be unlawful.
> > >
> > > Visit us at http://www.cognizant.com
> > >
>
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> prohibited and may be unlawful.
>
> Visit us at http://www.cognizant.com
>
Re: Nutch scoring algorithm
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
Hi,
Thanks for the explanation. I need some more info. I understand that
fieldNorm is byte-encoded normalization factor for the named field of
every document. This value is returned by norms(String field) of
SegmentReader class. Is this normalization factor calculated at index
time itself and is just read during searching?
I used explain to see boosts for different fields. Please see the
details below. As I see from explanation, "url" field is assigned a
boost of 4.0 and "anchor" field is assigned a boost of 2.0. Please
suggest me how I can alter boost values for different fields. Does it
need any configuration change during indexing itself?
page
* docNo = 1
* segment = 20050411183746
* digest = 3835653251e4598bee61618b1c64804c
* boost = 1.8572323
* lastModified = 1113224620000
* contentLength = 347
* primaryType = text
* subType = html
* url = http://localhost:8080/none.html
* title = None Document
score for query: none
* 1.5042199 = sum of:
o 0.4181689 = weight(url:none^4.0 in 1), product of:
+ 0.8728715 = queryWeight(url:none^4.0), product of:
# 4.0 = boost
# 1.9162908 = idf(docFreq=1)
# 0.11387514 = queryNorm
+ 0.4790727 = fieldWeight(url:none in 1), product of:
# 1.0 = tf(termFreq(url:none)=1)
# 1.9162908 = idf(docFreq=1)
# 0.25 = fieldNorm(field=url, doc=1)
o 1.0349152 = weight(anchor:none^2.0 in 1), product of:
+ 0.43643576 = queryWeight(anchor:none^2.0), product of:
# 2.0 = boost
# 1.9162908 = idf(docFreq=1)
# 0.11387514 = queryNorm
+ 2.3712888 = fieldWeight(anchor:none in 1), product of:
# 1.4142135 = tf(termFreq(anchor:none)=2)
# 1.9162908 = idf(docFreq=1)
# 0.875 = fieldNorm(field=anchor, doc=1)
o 0.05113577 = weight(content:none in 1), product of:
+ 0.21821788 = queryWeight(content:none), product of:
# 1.9162908 = idf(docFreq=1)
# 0.11387514 = queryNorm
+ 0.23433356 = fieldWeight(content:none in 1), product of:
# 2.236068 = tf(termFreq(content:none)=5)
# 1.9162908 = idf(docFreq=1)
# 0.0546875 = fieldNorm(field=content, doc=1)
Thanks,
Kannan
On Mon, 2005-04-11 at 17:47, Andy Liu wrote:
> fieldNorm is lengthNorm * document boost. The lengthNorm formula is
> defined within Lucene's similarity class (which is a function of the
> number of terms within the document) and the document boost is
> calculated in IndexSegment.java .
>
> Nutch assigns different boosts to each field so that you can tune your
> search results. For example, you can use explain to see if anchor
> matches are too strong, and adjust accordingly.
>
> Andy
>
> On Apr 11, 2005 12:17 AM, Kannan Sundaramoorthy
> <ka...@cognizant.com> wrote:
> >
> > Hi,
> > I am trying to understand how Nutch computes score for each document. I
> > could figure out how tf, idf and queryNorm are computed but I do not
> > understand how fieldNorm (normalisation for each field) value is
> > computed. It seems to be a magic number for me and this is where Nutch
> > seems to differ from Lucene in computing score.
> > Also Nutch assigns different boosts for different fields (e.g, 4.0 for
> > url field) and uses this value while computing queryWeight. Can anyone
> > explain these please?
> >
> > Thanks,
> > Kannan
> >
> > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> > If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> > Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> > prohibited and may be unlawful.
> >
> > Visit us at http://www.cognizant.com
> >
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com
Re: Nutch scoring algorithm
Posted by Andy Liu <an...@gmail.com>.
fieldNorm is lengthNorm * document boost. The lengthNorm formula is
defined within Lucene's similarity class (which is a function of the
number of terms within the document) and the document boost is
calculated in IndexSegment.java .
Nutch assigns different boosts to each field so that you can tune your
search results. For example, you can use explain to see if anchor
matches are too strong, and adjust accordingly.
Andy
On Apr 11, 2005 12:17 AM, Kannan Sundaramoorthy
<ka...@cognizant.com> wrote:
>
> Hi,
> I am trying to understand how Nutch computes score for each document. I
> could figure out how tf, idf and queryNorm are computed but I do not
> understand how fieldNorm (normalisation for each field) value is
> computed. It seems to be a magic number for me and this is where Nutch
> seems to differ from Lucene in computing score.
> Also Nutch assigns different boosts for different fields (e.g, 4.0 for
> url field) and uses this value while computing queryWeight. Can anyone
> explain these please?
>
> Thanks,
> Kannan
>
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> prohibited and may be unlawful.
>
> Visit us at http://www.cognizant.com
>
Nutch scoring algorithm
Posted by Kannan Sundaramoorthy <ka...@cognizant.com>.
Hi,
I am trying to understand how Nutch computes score for each document. I
could figure out how tf, idf and queryNorm are computed but I do not
understand how fieldNorm (normalisation for each field) value is
computed. It seems to be a magic number for me and this is where Nutch
seems to differ from Lucene in computing score.
Also Nutch assigns different boosts for different fields (e.g, 4.0 for
url field) and uses this value while computing queryWeight. Can anyone
explain these please?
Thanks,
Kannan
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Visit us at http://www.cognizant.com