You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Apache Lucene <lu...@gmail.com> on 2006/10/03 22:42:26 UTC
focussed crawling
I would like to setup a focussed crawler using Nutch. Assuming I have a way
to detect which page is relevant to the topic under consideration what is
the best architecture? Here are the constraints for the crawler.
(1) The first crawl should result in links/pages which are related to the
domain.
(2) The non-relevant documents in the database should be within N hops from
the relevant document. The assumption is a non-relevant document closer to
the relevant node in the web graph might contain links to other relevant
documents.
(1) should be fairly straight forward however I am not sure what it involves
to implement (2). I am also not sure how the PageRank will work in case of a
focussed crawler.
Any comments? suggestions?
Re: focussed crawling
Posted by Alvaro Cabrerizo <to...@gmail.com>.
Although I havent use it. After making a crawl, at least in nutch 0.8,
you can make a "./bin/nutch/mergedb outputdb your_db -filter". Using
the filter option you can generate a new db, filtering links you wanna
remove. And use it to make a recrawl.
Hope it helps.
Re: focussed crawling
Posted by Apache Lucene <lu...@gmail.com>.
The problem with keeping everything is, when I do a recrawl I dont want to
process all the links. The number of relevant links might be negligible
compared to the number of irrelevant links.
On 10/4/06, Jim Wilson <wi...@gmail.com> wrote:
>
> It seems like you can probably keep everything in the Links and
> Pages/Content - as long as it doesn't end up in the Index, right? If so,
> you may only need to modify the indexer and leave the rest alone.
>
> (I may not understand the problem though, and I'm certainly not a Nutch
> expert - just trying to help :)
>
> -- Jim
>
> On 10/4/06, Apache Lucene <lu...@gmail.com> wrote:
> >
> > On 10/3/06, Jim Wilson <wi...@gmail.com> wrote:
> > >
> > > PageRank is a Trademark of Google, and a source of great revenue for
> > them
> > > -
> > > you'll have to call it something else. :(
> >
> >
> > I was referring to the link score nutch assigns.
> >
> > Determining whether a page is relevant to a topic (with any degree of
> > > accuracy) is a harder problem that it may appear - though your opening
> > > post
> > > says to assume you have a way to do it. For example, suppose the word
> > > "buffalo" appears on a page. Does this mean the animal, or the NFL
> > team,
> > > the city, or the spicy sauce?
> >
> >
> > My case is fairly simple. The case you are mentioning is definitely a
> > harder
> > problem.
> >
> > Another concern is the assumption that documents linked-to by relevant
> > > documents are themselves at all relevant. Take Wikipedia for example
> -
> > > there are lots of links on every page that have nothing to do with the
> > > article (such as Main Page, Community Portal, Privacy Policy,
> etc). If
> > N
> > > is
> > > any more than 1 or 2, you'll probably be swamped with non-relevant
> pages
> >
> >
> > The reason I wanted this architecture was to get as many links as
> > possible.
> > The link database would have the non-relevant links however the final
> > index
> > would not. The non-relevant links will be used only for getting the
> links
> > for more relevant links (if any). At the end of first crawl the database
> > will have the following:
> >
> > Links: Relevant links + Small number of non-relevant links
> > Pages/Content: Relevant pages only
> > Index: Relevant pages only
> >
> > In the subsequent recrawling I would just have to visit recrawl a small
> > subset as opposed to the entire links had I stored all the links. If I
> > dont
> > store the non-relevant links then I might not be able to get the
> relevant
> > links which might be in the non-relevant pages. I am willing to have
> false
> > positives in the database but keeping them low by setting N
> appropriately.
> > I
> > hope this makes sense. Maybe the term "focussed crawling" was wrong..:)
> >
> >
> > > .In researching the problem, you might want to check out the Carrot
> > > Clustering Engine (http://demo.carrot-search.com/carrot2-webapp/main
> > ). It
> > > may do what you want OOTB.
> >
> >
> > If I am right, Carrot is useful in clustering the search results and
> > helpful
> > in crawling. I will take a fresh look at it for crawling.
> >
> >
>
>
Re: focussed crawling
Posted by Jim Wilson <wi...@gmail.com>.
It seems like you can probably keep everything in the Links and
Pages/Content - as long as it doesn't end up in the Index, right? If so,
you may only need to modify the indexer and leave the rest alone.
(I may not understand the problem though, and I'm certainly not a Nutch
expert - just trying to help :)
-- Jim
On 10/4/06, Apache Lucene <lu...@gmail.com> wrote:
>
> On 10/3/06, Jim Wilson <wi...@gmail.com> wrote:
> >
> > PageRank is a Trademark of Google, and a source of great revenue for
> them
> > -
> > you'll have to call it something else. :(
>
>
> I was referring to the link score nutch assigns.
>
> Determining whether a page is relevant to a topic (with any degree of
> > accuracy) is a harder problem that it may appear - though your opening
> > post
> > says to assume you have a way to do it. For example, suppose the word
> > "buffalo" appears on a page. Does this mean the animal, or the NFL
> team,
> > the city, or the spicy sauce?
>
>
> My case is fairly simple. The case you are mentioning is definitely a
> harder
> problem.
>
> Another concern is the assumption that documents linked-to by relevant
> > documents are themselves at all relevant. Take Wikipedia for example -
> > there are lots of links on every page that have nothing to do with the
> > article (such as Main Page, Community Portal, Privacy Policy, etc). If
> N
> > is
> > any more than 1 or 2, you'll probably be swamped with non-relevant pages
>
>
> The reason I wanted this architecture was to get as many links as
> possible.
> The link database would have the non-relevant links however the final
> index
> would not. The non-relevant links will be used only for getting the links
> for more relevant links (if any). At the end of first crawl the database
> will have the following:
>
> Links: Relevant links + Small number of non-relevant links
> Pages/Content: Relevant pages only
> Index: Relevant pages only
>
> In the subsequent recrawling I would just have to visit recrawl a small
> subset as opposed to the entire links had I stored all the links. If I
> dont
> store the non-relevant links then I might not be able to get the relevant
> links which might be in the non-relevant pages. I am willing to have false
> positives in the database but keeping them low by setting N appropriately.
> I
> hope this makes sense. Maybe the term "focussed crawling" was wrong..:)
>
>
> > .In researching the problem, you might want to check out the Carrot
> > Clustering Engine (http://demo.carrot-search.com/carrot2-webapp/main
> ). It
> > may do what you want OOTB.
>
>
> If I am right, Carrot is useful in clustering the search results and
> helpful
> in crawling. I will take a fresh look at it for crawling.
>
>
Re: focussed crawling
Posted by Apache Lucene <lu...@gmail.com>.
On 10/3/06, Jim Wilson <wi...@gmail.com> wrote:
>
> PageRank is a Trademark of Google, and a source of great revenue for them
> -
> you'll have to call it something else. :(
I was referring to the link score nutch assigns.
Determining whether a page is relevant to a topic (with any degree of
> accuracy) is a harder problem that it may appear - though your opening
> post
> says to assume you have a way to do it. For example, suppose the word
> "buffalo" appears on a page. Does this mean the animal, or the NFL team,
> the city, or the spicy sauce?
My case is fairly simple. The case you are mentioning is definitely a harder
problem.
Another concern is the assumption that documents linked-to by relevant
> documents are themselves at all relevant. Take Wikipedia for example -
> there are lots of links on every page that have nothing to do with the
> article (such as Main Page, Community Portal, Privacy Policy, etc). If N
> is
> any more than 1 or 2, you'll probably be swamped with non-relevant pages
The reason I wanted this architecture was to get as many links as possible.
The link database would have the non-relevant links however the final index
would not. The non-relevant links will be used only for getting the links
for more relevant links (if any). At the end of first crawl the database
will have the following:
Links: Relevant links + Small number of non-relevant links
Pages/Content: Relevant pages only
Index: Relevant pages only
In the subsequent recrawling I would just have to visit recrawl a small
subset as opposed to the entire links had I stored all the links. If I dont
store the non-relevant links then I might not be able to get the relevant
links which might be in the non-relevant pages. I am willing to have false
positives in the database but keeping them low by setting N appropriately. I
hope this makes sense. Maybe the term "focussed crawling" was wrong..:)
> .In researching the problem, you might want to check out the Carrot
> Clustering Engine (http://demo.carrot-search.com/carrot2-webapp/main). It
> may do what you want OOTB.
If I am right, Carrot is useful in clustering the search results and helpful
in crawling. I will take a fresh look at it for crawling.
Re: focussed crawling
Posted by Jim Wilson <wi...@gmail.com>.
PageRank is a Trademark of Google, and a source of great revenue for them -
you'll have to call it something else. :(
Determining whether a page is relevant to a topic (with any degree of
accuracy) is a harder problem that it may appear - though your opening post
says to assume you have a way to do it. For example, suppose the word
"buffalo" appears on a page. Does this mean the animal, or the NFL team,
the city, or the spicy sauce?
Another concern is the assumption that documents linked-to by relevant
documents are themselves at all relevant. Take Wikipedia for example -
there are lots of links on every page that have nothing to do with the
article (such as Main Page, Community Portal, Privacy Policy, etc). If N is
any more than 1 or 2, you'll probably be swamped with non-relevant pages.
In researching the problem, you might want to check out the Carrot
Clustering Engine (http://demo.carrot-search.com/carrot2-webapp/main). It
may do what you want OOTB.
-- Jim
On 10/3/06, Apache Lucene <lu...@gmail.com> wrote:
>
> I would like to setup a focussed crawler using Nutch. Assuming I have a
> way
> to detect which page is relevant to the topic under consideration what is
> the best architecture? Here are the constraints for the crawler.
>
> (1) The first crawl should result in links/pages which are related to the
> domain.
> (2) The non-relevant documents in the database should be within N hops
> from
> the relevant document. The assumption is a non-relevant document closer to
> the relevant node in the web graph might contain links to other relevant
> documents.
>
> (1) should be fairly straight forward however I am not sure what it
> involves
> to implement (2). I am also not sure how the PageRank will work in case of
> a
> focussed crawler.
>
> Any comments? suggestions?
>
>