You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Rob Pettengill <ro...@earthlink.net> on 2005/07/08 16:32:36 UTC
indexing workflow, refetching, and missing links questions + a nutch subcommand reference
I am crawling files for a vertical search engine - a case which has
similarities both to the intranet search with CrawlTool and the
internet search workflow in the tutorial.
I have two sets of questions that may be related:
1. In CrawlTool after all the segments are fetched, there is a
comment which reads:
// Re-fetch everything to get the complete set of incoming
anchor texts
// associated with each page. We should fix this, so that
we can update
// the previously fetched segments with the anchors that are
now in the
// database, but until that algorithm is written, we re-fetch.
Then there are calls to FetchListTool and Fetcher which creates a new
segment with all of the documents and refetches them, before indexing
the new combined segment.
In the internet indexing work flow described in the tutorial there
are no equivalent steps to the refetching steps in the CrawlTool.
Can someone explain what is going on here?
====
2. I ended up with a half dozen segments which I combined into one
new segment using the SegmentMergeTool. Inspired by the workflow
differences described above, as an experiment
I built a new page and link db via a workflow like this:
nutch admin db2 create
nutch updatedb db2 combined segment.
I then compared the orignial db with db2 using WebDBReader with the -
stats option. What I found was that db2 only had about half as many
links as the original db.
Where did the links go? Doesn't the combined segment have all of the
original content?
Do these results correctly imply that a refetch is necessary any time
segments are combined? If so why?
====
P.S.
As I've been sorting out Nutch, I found myself constantly going back
and forth between the Java docs, the java source code, the nutch
script code, and the tutorial. I ended up putting together a
document that summarized the key information extracted from all of
the above for commands that can be issued through the nutch script.
It can be found at:
http://mesavida.com/NutchSubCmds.html
Do you all think that this kind of nutch subcommand reference is
useful? If so where should I put it?
Thanks,
Rob
--
Rob Pettengill,
Questions about petroleum?
Goto: http://AskAboutOil.com/
Re: indexing workflow, refetching, and missing links questions + a nutch subcommand reference
Posted by Rob Pettengill <ro...@earthlink.net>.
Thanks for the cogent explanation!
I'm guessing that adding the link info to the index of earlier pages
from pages fetched later in other segments is awkward because they
are not in the index being currently built.
This still leaves me a bit puzzled as to how I could merge segments,
rebuild the page and link data base from the merged segment and end
up with roughly half the number of links in the new db. It seems
that there should be no change in the number of links. I'm beginning
to suspect pilot error of some kind on my part.
Thanks,
Rob
On 2005, Jul 12, at 8:05 AM, Russell Mayor wrote:
> On 7/8/05, Rob Pettengill <ro...@earthlink.net> wrote:
>
>> 1. In CrawlTool after all the segments are fetched, there is a
>> comment which reads:
>>
>> // Re-fetch everything to get the complete set of incoming
>> anchor texts
>> // associated with each page. We should fix this, so that
>> we can update
>> // the previously fetched segments with the anchors that are
>> now in the
>> // database, but until that algorithm is written, we re-
>> fetch.
>>
>> Then there are calls to FetchListTool and Fetcher which creates a new
>> segment with all of the documents and refetches them, before indexing
>> the new combined segment.
>>
>> In the internet indexing work flow described in the tutorial there
>> are no equivalent steps to the refetching steps in the CrawlTool.
>>
>> Can someone explain what is going on here?
>>
>>
>
> For each page in the page database nutch tracks the anchor text for
> links to that page (that is, the text of the links on other pages that
> point to that page). This anchor text forms one of the fields that is
> eventually output into the Lucene indexes that the front-end searches.
> (Anchor text is useful in this way because it is often a good, short
> description of the page being linked to).
>
> The CrawlTool works by iteratively fetching a set of injected pages,
> the pages that they link to, the pages that they link to etc, creating
> a segment for each iteration. However, the earlier segments are
> created before the later pages are discovered, and so don't contain
> the anchor text for all the links on those later pages that link to
> pages in the earler segments. This is becasue, at the end of fetching
> a fetch list, the page database is updated with all the new link
> information found during the fetch, but previously generated segments
> are left untouched.
>
> The code you have seen in the CrawlTool (which just removes all the
> existing segments and refetches all the pages in the entire page
> database as one large segment) is a hack so that the final segment
> contains the correct anchor text data for each page. It does this at
> the expense of effectively fetching every page twice.
>
> Note that the problem of poor anchor text goes away once pages start
> to go stale and are refetched - the CrawlTool only performs as you
> have indicated because it is not intended to be used continuously, I
> beleive.
>
> Also, the CrawlTool has been changed slightly in the current
> development version of nutch so that it does not refetch all the pages
> again. Instead, the previously fetched segments are updated in place.
> This is much more efficient.
>
Re: indexing workflow, refetching, and missing links questions + a nutch subcommand reference
Posted by Russell Mayor <ru...@gmail.com>.
On 7/8/05, Rob Pettengill <ro...@earthlink.net> wrote:
> 1. In CrawlTool after all the segments are fetched, there is a
> comment which reads:
>
> // Re-fetch everything to get the complete set of incoming
> anchor texts
> // associated with each page. We should fix this, so that
> we can update
> // the previously fetched segments with the anchors that are
> now in the
> // database, but until that algorithm is written, we re-fetch.
>
> Then there are calls to FetchListTool and Fetcher which creates a new
> segment with all of the documents and refetches them, before indexing
> the new combined segment.
>
> In the internet indexing work flow described in the tutorial there
> are no equivalent steps to the refetching steps in the CrawlTool.
>
> Can someone explain what is going on here?
>
For each page in the page database nutch tracks the anchor text for
links to that page (that is, the text of the links on other pages that
point to that page). This anchor text forms one of the fields that is
eventually output into the Lucene indexes that the front-end searches.
(Anchor text is useful in this way because it is often a good, short
description of the page being linked to).
The CrawlTool works by iteratively fetching a set of injected pages,
the pages that they link to, the pages that they link to etc, creating
a segment for each iteration. However, the earlier segments are
created before the later pages are discovered, and so don't contain
the anchor text for all the links on those later pages that link to
pages in the earler segments. This is becasue, at the end of fetching
a fetch list, the page database is updated with all the new link
information found during the fetch, but previously generated segments
are left untouched.
The code you have seen in the CrawlTool (which just removes all the
existing segments and refetches all the pages in the entire page
database as one large segment) is a hack so that the final segment
contains the correct anchor text data for each page. It does this at
the expense of effectively fetching every page twice.
Note that the problem of poor anchor text goes away once pages start
to go stale and are refetched - the CrawlTool only performs as you
have indicated because it is not intended to be used continuously, I
beleive.
Also, the CrawlTool has been changed slightly in the current
development version of nutch so that it does not refetch all the pages
again. Instead, the previously fetched segments are updated in place.
This is much more efficient.
Re: [Nutch-general] indexing workflow, refetching, and missing links
questions + a nutch subcommand reference
Posted by Matthias Jaekle <ja...@eventax.de>.
> I also hope Matthias forgives me for any violence that I did to his
> concept of how this part of the Wiki should be organized.
Sure. Whatever you make to get a better wiki is fine.
So, make it better. Expand it. Keep it up-to-date.
The stuff I made is over one year old and maybe incorrect in some points
now.
Matthias
Re: [Nutch-general] indexing workflow, refetching, and missing links questions + a nutch subcommand reference
Posted by Rob Pettengill <ro...@earthlink.net>.
I took a first pass at merging the stuff I had into http://
wiki.apache.org/nutch/CommandLineOptions
The Wiki syntax is pretty funky especially with mixed case Java class
names. I also tried to make things like the usage examples a bit
more consistent, so I'd appreciate some fresh eyes making sure that
things ended up correct.
I also hope Matthias forgives me for any violence that I did to his
concept of how this part of the Wiki should be organized.
--
Robert C. Pettengill, Ph.D.
rcp@stanfordalumni.org
Questions about petroleum?
Goto: http://AskAboutOil.com/
Need help implementing search?
Goto: http://MesaVida.com/
On 2005, Jul 16, at 10:25 AM, <og...@yahoo.com> <ogjunk-
nutch@yahoo.com> wrote:
> Hi Rob,
>
> That's a useful Nutch command reference you have there. Could you
> please put it on Nutch's Wiki? http://wiki.apache.org/nutch/
>
> Maybe somewhere along/in/linked from
> http://wiki.apache.org/nutch/CommandLineOptions page
>
> Otis
>
>
> --- Rob Pettengill <ro...@earthlink.net> wrote:
>
>
Re: [Nutch-general] indexing workflow, refetching, and missing links questions + a nutch subcommand reference
Posted by og...@yahoo.com.
Hi Rob,
That's a useful Nutch command reference you have there. Could you
please put it on Nutch's Wiki? http://wiki.apache.org/nutch/
Maybe somewhere along/in/linked from
http://wiki.apache.org/nutch/CommandLineOptions page
Otis
--- Rob Pettengill <ro...@earthlink.net> wrote:
> I am crawling files for a vertical search engine - a case which has
> similarities both to the intranet search with CrawlTool and the
> internet search workflow in the tutorial.
>
> I have two sets of questions that may be related:
>
> 1. In CrawlTool after all the segments are fetched, there is a
> comment which reads:
>
> // Re-fetch everything to get the complete set of incoming
> anchor texts
> // associated with each page. We should fix this, so that
> we can update
> // the previously fetched segments with the anchors that are
>
> now in the
> // database, but until that algorithm is written, we
> re-fetch.
>
> Then there are calls to FetchListTool and Fetcher which creates a new
>
> segment with all of the documents and refetches them, before indexing
>
> the new combined segment.
>
> In the internet indexing work flow described in the tutorial there
> are no equivalent steps to the refetching steps in the CrawlTool.
>
> Can someone explain what is going on here?
>
> ====
>
> 2. I ended up with a half dozen segments which I combined into one
> new segment using the SegmentMergeTool. Inspired by the workflow
> differences described above, as an experiment
> I built a new page and link db via a workflow like this:
> nutch admin db2 create
> nutch updatedb db2 combined segment.
>
> I then compared the orignial db with db2 using WebDBReader with the -
>
> stats option. What I found was that db2 only had about half as many
>
> links as the original db.
>
> Where did the links go? Doesn't the combined segment have all of the
>
> original content?
>
> Do these results correctly imply that a refetch is necessary any time
>
> segments are combined? If so why?
>
> ====
> P.S.
>
> As I've been sorting out Nutch, I found myself constantly going back
>
> and forth between the Java docs, the java source code, the nutch
> script code, and the tutorial. I ended up putting together a
> document that summarized the key information extracted from all of
> the above for commands that can be issued through the nutch script.
>
> It can be found at:
>
> http://mesavida.com/NutchSubCmds.html
>
> Do you all think that this kind of nutch subcommand reference is
> useful? If so where should I put it?
>
> Thanks,
> Rob
>
>
> --
> Rob Pettengill,
>
> Questions about petroleum?
> Goto: http://AskAboutOil.com/
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the 'Do More With Dual!' webinar
> happening
> July 14 at 8am PDT/11am EDT. We invite you to explore the latest in
> dual
> core and dual graphics technology at this free one hour event hosted
> by HP,
> AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
>