You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "J. Gobel" <jj...@gmail.com> on 2013/01/13 11:14:19 UTC
nutch 2.x recrawl re-crawl
hi there,
I am trying to figure out what the best method is to recrawl certain sites.
I am crawling news-sites and they update their frontpage quite often, so I
need o crawl their frontpage/index.php etc. often and have Nutch fetch the
new links + content.
I cannot find an answer to my question in the mailing archive. I also
checked other websites one of them is quite good in explaining
http://pascaldimassimo.com/2010/06/11/how-to-re-crawl-with-nutch/
This is discussed on : How to recrawle
nutch<http://stackoverflow.com/questions/13873694/how-to-recrawle-nutch>
I followed the instructions found on that website, and changed the
db.interval to start with a re-fetch after 24 hours. Hoping that the
mechanism of Nutch will work and assign websites that have static pages a
higher fetch number, so that eventually it will be fetched once every 90
days. And those pages that update often, will be fetched multiple times per
day.
At the moment I am testing if this works.
I wonder though, once this set-up works and I start to crawl more news
sites all of the pages will be fetched every 24 hours, and according to the
rules it will be fetched later or earlier from that moment on. This is not
desirable as this means that ALL urls will be fetched daily. Is there some
way that I can assign priority to certain pages. For example, give all
index.php/htm/html pages a fetch interval of 60 minutes, and all other
pages 3 days?
I use Nutch 2.1 with Mysql backend on Ubuntu 12.04
Thanks in advance,
J
Re: nutch 2.x recrawl re-crawl
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Jaap,
On Tue, Jan 15, 2013 at 1:52 AM, J. Gobel <jj...@gmail.com> wrote:
> http://www.latestnews.nl/ nutch.score=10 nutch.fetchInterval=60
> userType=open_source
>
> (what is userType?)
>
AFAIK this can be any metadata field which you then assign a value. In the
example here we use userType to refer to the nature of Nutch as an open
source project. I've updated with wiki with this, thanks for pointing it
out.
>
> Now I wonder, I am not completely familiar how Nutch is fetching the urls.
> If I inject the urls/seed.txt on every run, Nutch will fetch the urls from
> index.php . but will it also parse them on the same run?
Nutch will fetch and parse the first round of URLs in your seed.txt and
will extract the outlinks based on the -topN parameter for the next round
of fetching and subsequent parsing.
> Or will Nutch
> parse previous fetched Urls?
If URLs are due for a fetch, then when you generate Nutch attempts to fetch
them. Once they have been fetched, they should be parsed.
How does this work, and how can I influence
> what urls will be fetched?
>
Well you can start with assigning a custom scoring value and fetching
interval to the injected URLs.
>
> In my humble opinion, this class should be mentioned in the recrawl wiki.
>
If you would like to contribute to the wiki documentation then please sign
up to the wiki and I can grant you the necessary karma.
>
> Lewis
Re: nutch 2.x recrawl re-crawl
Posted by "J. Gobel" <jj...@gmail.com>.
Hi,
I stumbled on http://wiki.apache.org/nutch/bin/nutch_inject
I can indicate in my seeds.txt file per url what the fetchtime and scoring
should be. This is great :-)
My seeds.txt now contains , tab delimited.
http://www.latestnews.nl/ nutch.score=10 nutch.fetchInterval=60
userType=open_source
(what is userType?)
Now I wonder, I am not completely familiar how Nutch is fetching the urls.
If I inject the urls/seed.txt on every run, Nutch will fetch the urls from
index.php . but will it also parse them on the same run? Or will Nutch
parse previous fetched Urls? How does this work, and how can I influence
what urls will be fetched?
In my humble opinion, this class should be mentioned in the recrawl wiki.
Thanks in advance,
Jaap
On Tue, Jan 15, 2013 at 1:43 AM, <al...@aim.com> wrote:
> I think there is no need to a new plugin or something like that. If you
> know list of news urls you need to inject them each cycle in order to fetch
> them and their new inlinks, since when you inject a url its fetch time is
> set to the current time.
>
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: J. Gobel <jj...@gmail.com>
> To: user <us...@nutch.apache.org>
> Sent: Mon, Jan 14, 2013 2:47 pm
> Subject: Re: nutch 2.x recrawl re-crawl
>
>
> I looked at the freegenerator tool.. But I don't see how I can use that
> tool to crawl www.thelatestnews.com/index.php , scrape the newest links
> from that page, and fetch those.
>
> My dream plugin would be to allow me to feed Nutch priority urls such as ,
> '
> www.thelatestnews.com/index.php' , and that it will fetch all the new
> links
> found on that url with a priority.
>
>
>
> On Mon, Jan 14, 2013 at 11:42 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> > freegenerator tool
>
>
>
Re: nutch 2.x recrawl re-crawl
Posted by al...@aim.com.
I think there is no need to a new plugin or something like that. If you know list of news urls you need to inject them each cycle in order to fetch them and their new inlinks, since when you inject a url its fetch time is set to the current time.
Alex.
-----Original Message-----
From: J. Gobel <jj...@gmail.com>
To: user <us...@nutch.apache.org>
Sent: Mon, Jan 14, 2013 2:47 pm
Subject: Re: nutch 2.x recrawl re-crawl
I looked at the freegenerator tool.. But I don't see how I can use that
tool to crawl www.thelatestnews.com/index.php , scrape the newest links
from that page, and fetch those.
My dream plugin would be to allow me to feed Nutch priority urls such as , '
www.thelatestnews.com/index.php' , and that it will fetch all the new links
found on that url with a priority.
On Mon, Jan 14, 2013 at 11:42 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> freegenerator tool
Re: nutch 2.x recrawl re-crawl
Posted by "J. Gobel" <jj...@gmail.com>.
I looked at the freegenerator tool.. But I don't see how I can use that
tool to crawl www.thelatestnews.com/index.php , scrape the newest links
from that page, and fetch those.
My dream plugin would be to allow me to feed Nutch priority urls such as , '
www.thelatestnews.com/index.php' , and that it will fetch all the new links
found on that url with a priority.
On Mon, Jan 14, 2013 at 11:42 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> freegenerator tool
RE: nutch 2.x recrawl re-crawl
Posted by Markus Jelsma <ma...@openindex.io>.
Hi
The fetch schedule should be the same code as in 1.x but i'm not too sure about the generator but it should be easy to port. An alternative is to use the freegenerator tool to force specific records to be fetched.
Cheers
-----Original message-----
> From:J. Gobel <jj...@gmail.com>
> Sent: Mon 14-Jan-2013 23:31
> To: user@nutch.apache.org
> Subject: Re: nutch 2.x recrawl re-crawl
>
> Hi Markus,
>
> Yes, but sadly it is not available for Nutch 2.x :( .. My work around is to
> start fetching all urls every 24 hours and see how that goes. Alternatively
> I could change the fetchinterval field in the MySQL tables. But I rather
> have something functional without altering anything directly in MYSQL.
>
> rgds,
> Jaap
>
>
> On Mon, Jan 14, 2013 at 11:03 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> > Hi,
> >
> > Please see NUTCH-1388, i added it to 1.6. It allows you to inject records
> > with a fixed fetch interval regardless of whatever the adaptive schedule
> > would make of it. It's ideally to bootstrap a crawl and have it crawl a
> > news listing or whatever on a regular basis.
> >
> > Cheers,
> >
> > -----Original message-----
> > > From:Lewis John Mcgibbney <le...@gmail.com>
> > > Sent: Mon 14-Jan-2013 21:34
> > > To: user@nutch.apache.org
> > > Subject: Re: nutch 2.x recrawl re-crawl
> > >
> > > Hi,
> > >
> > > On Mon, Jan 14, 2013 at 3:56 AM, J. Gobel <jj...@gmail.com> wrote:
> > >
> > > > Hi Lewis,
> > > >
> > > > Thanks for your mail.
> > > >
> > > > My ideal goal would be to crawl the index.php several times per day,
> > and
> > > > fetch the new urls from that page and parse them. Then I know 'for
> > sure'
> > > > that my index is up to date.
> > > >
> > >
> > > Sounds like this fetching schedule is your cup of tea then. Unless of
> > > course anyone has a better more efficient idea?
> > > Either this, or you could run your crawls with different url filter files
> > > (tailored to filter non desired mime types) depending on how often you
> > > envisage them to change.
> > >
> > >
> > > >
> > > > You mentioned AdaptiveFetchSchedule , but I am no programmer. Is there
> > > > perhaps any other way to do the above with configuring Nutch 2.1?
> > > >
> > > > Ah ok. Well to use the MimeAdaptiveFetchSchule, it would need to be
> > ported
> > > to 2.x... which requires coding.
> > >
> > > Maybe you can try the above suggestion to see if this works for your
> > needs.
> > > If not, then porting the fetch schedule to 2.x may be the best way to go.
> > > Lewis
> > >
> >
>
Re: nutch 2.x recrawl re-crawl
Posted by "J. Gobel" <jj...@gmail.com>.
Hi Markus,
Yes, but sadly it is not available for Nutch 2.x :( .. My work around is to
start fetching all urls every 24 hours and see how that goes. Alternatively
I could change the fetchinterval field in the MySQL tables. But I rather
have something functional without altering anything directly in MYSQL.
rgds,
Jaap
On Mon, Jan 14, 2013 at 11:03 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> Hi,
>
> Please see NUTCH-1388, i added it to 1.6. It allows you to inject records
> with a fixed fetch interval regardless of whatever the adaptive schedule
> would make of it. It's ideally to bootstrap a crawl and have it crawl a
> news listing or whatever on a regular basis.
>
> Cheers,
>
> -----Original message-----
> > From:Lewis John Mcgibbney <le...@gmail.com>
> > Sent: Mon 14-Jan-2013 21:34
> > To: user@nutch.apache.org
> > Subject: Re: nutch 2.x recrawl re-crawl
> >
> > Hi,
> >
> > On Mon, Jan 14, 2013 at 3:56 AM, J. Gobel <jj...@gmail.com> wrote:
> >
> > > Hi Lewis,
> > >
> > > Thanks for your mail.
> > >
> > > My ideal goal would be to crawl the index.php several times per day,
> and
> > > fetch the new urls from that page and parse them. Then I know 'for
> sure'
> > > that my index is up to date.
> > >
> >
> > Sounds like this fetching schedule is your cup of tea then. Unless of
> > course anyone has a better more efficient idea?
> > Either this, or you could run your crawls with different url filter files
> > (tailored to filter non desired mime types) depending on how often you
> > envisage them to change.
> >
> >
> > >
> > > You mentioned AdaptiveFetchSchedule , but I am no programmer. Is there
> > > perhaps any other way to do the above with configuring Nutch 2.1?
> > >
> > > Ah ok. Well to use the MimeAdaptiveFetchSchule, it would need to be
> ported
> > to 2.x... which requires coding.
> >
> > Maybe you can try the above suggestion to see if this works for your
> needs.
> > If not, then porting the fetch schedule to 2.x may be the best way to go.
> > Lewis
> >
>
RE: nutch 2.x recrawl re-crawl
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
Please see NUTCH-1388, i added it to 1.6. It allows you to inject records with a fixed fetch interval regardless of whatever the adaptive schedule would make of it. It's ideally to bootstrap a crawl and have it crawl a news listing or whatever on a regular basis.
Cheers,
-----Original message-----
> From:Lewis John Mcgibbney <le...@gmail.com>
> Sent: Mon 14-Jan-2013 21:34
> To: user@nutch.apache.org
> Subject: Re: nutch 2.x recrawl re-crawl
>
> Hi,
>
> On Mon, Jan 14, 2013 at 3:56 AM, J. Gobel <jj...@gmail.com> wrote:
>
> > Hi Lewis,
> >
> > Thanks for your mail.
> >
> > My ideal goal would be to crawl the index.php several times per day, and
> > fetch the new urls from that page and parse them. Then I know 'for sure'
> > that my index is up to date.
> >
>
> Sounds like this fetching schedule is your cup of tea then. Unless of
> course anyone has a better more efficient idea?
> Either this, or you could run your crawls with different url filter files
> (tailored to filter non desired mime types) depending on how often you
> envisage them to change.
>
>
> >
> > You mentioned AdaptiveFetchSchedule , but I am no programmer. Is there
> > perhaps any other way to do the above with configuring Nutch 2.1?
> >
> > Ah ok. Well to use the MimeAdaptiveFetchSchule, it would need to be ported
> to 2.x... which requires coding.
>
> Maybe you can try the above suggestion to see if this works for your needs.
> If not, then porting the fetch schedule to 2.x may be the best way to go.
> Lewis
>
Re: nutch 2.x recrawl re-crawl
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,
On Mon, Jan 14, 2013 at 3:56 AM, J. Gobel <jj...@gmail.com> wrote:
> Hi Lewis,
>
> Thanks for your mail.
>
> My ideal goal would be to crawl the index.php several times per day, and
> fetch the new urls from that page and parse them. Then I know 'for sure'
> that my index is up to date.
>
Sounds like this fetching schedule is your cup of tea then. Unless of
course anyone has a better more efficient idea?
Either this, or you could run your crawls with different url filter files
(tailored to filter non desired mime types) depending on how often you
envisage them to change.
>
> You mentioned AdaptiveFetchSchedule , but I am no programmer. Is there
> perhaps any other way to do the above with configuring Nutch 2.1?
>
> Ah ok. Well to use the MimeAdaptiveFetchSchule, it would need to be ported
to 2.x... which requires coding.
Maybe you can try the above suggestion to see if this works for your needs.
If not, then porting the fetch schedule to 2.x may be the best way to go.
Lewis
Re: nutch 2.x recrawl re-crawl
Posted by "J. Gobel" <jj...@gmail.com>.
Hi Lewis,
Thanks for your mail.
My ideal goal would be to crawl the index.php several times per day, and
fetch the new urls from that page and parse them. Then I know 'for sure'
that my index is up to date.
You mentioned AdaptiveFetchSchedule , but I am no programmer. Is there
perhaps any other way to do the above with configuring Nutch 2.1?
Thanks in advance,
Jaap
On Mon, Jan 14, 2013 at 12:45 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi J,
>
> On Sun, Jan 13, 2013 at 2:14 AM, J. Gobel <jj...@gmail.com> wrote:
>
> >
> > At the moment I am testing if this works.
> >
>
> Please keep us updated then.
>
> >
> > This is not
> > desirable as this means that ALL urls will be fetched daily.
>
>
> Typically if URLs are dynamically changing, you would want to maintain a
> webdb of the most recent version of such a page... no? This is why Nutch is
> designed to scale well.
>
>
> > Is there some
> > way that I can assign priority to certain pages. For example, give all
> > index.php/htm/html pages a fetch interval of 60 minutes, and all other
> > pages 3 days?
> >
> > Markus, implemented an extension of the AdaptiveFetchSchedule [0] which
> allows you to specify a configuration file [1] containing the mime-types
> and thier inc and dec rates based upon your preference.
>
> I acknowledge that you are on 2.x, but maybe attempting a port to 2.x would
> be your best option.
>
> hth
> Lewis
>
> [0]
>
> http://svn.apache.org/repos/asf/nutch/trunk/src/java/org/apache/nutch/crawl/MimeAdaptiveFetchSchedule.java
> [1]
> http://svn.apache.org/repos/asf/nutch/trunk/conf/adaptive-mimetypes.txt
>
Re: nutch 2.x recrawl re-crawl
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Bayu,
Yes it will run fine on 1.6.
Lewis
On Sun, Jan 13, 2013 at 10:24 PM, Bayu Widyasanyata <bwidyasanyata@gmail.com
> wrote:
> On Mon, Jan 14, 2013 at 6:45 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > > Markus, implemented an extension of the AdaptiveFetchSchedule [0] which
> > allows you to specify a configuration file [1] containing the mime-types
> > and thier inc and dec rates based upon your preference.
> >
> > I acknowledge that you are on 2.x, but maybe attempting a port to 2.x
> would
> > be your best option.
> >
>
> Hi,
>
> Is there any documentation/URLs implement this extension on nutch?
> Does this ext well running on nutch 1.6?
>
> --
> wassalam,
> [bayu]
>
--
*Lewis*
Re: nutch 2.x recrawl re-crawl
Posted by Bayu Widyasanyata <bw...@gmail.com>.
On Mon, Jan 14, 2013 at 6:45 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> > Markus, implemented an extension of the AdaptiveFetchSchedule [0] which
> allows you to specify a configuration file [1] containing the mime-types
> and thier inc and dec rates based upon your preference.
>
> I acknowledge that you are on 2.x, but maybe attempting a port to 2.x would
> be your best option.
>
Hi,
Is there any documentation/URLs implement this extension on nutch?
Does this ext well running on nutch 1.6?
--
wassalam,
[bayu]
Re: nutch 2.x recrawl re-crawl
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi J,
On Sun, Jan 13, 2013 at 2:14 AM, J. Gobel <jj...@gmail.com> wrote:
>
> At the moment I am testing if this works.
>
Please keep us updated then.
>
> This is not
> desirable as this means that ALL urls will be fetched daily.
Typically if URLs are dynamically changing, you would want to maintain a
webdb of the most recent version of such a page... no? This is why Nutch is
designed to scale well.
> Is there some
> way that I can assign priority to certain pages. For example, give all
> index.php/htm/html pages a fetch interval of 60 minutes, and all other
> pages 3 days?
>
> Markus, implemented an extension of the AdaptiveFetchSchedule [0] which
allows you to specify a configuration file [1] containing the mime-types
and thier inc and dec rates based upon your preference.
I acknowledge that you are on 2.x, but maybe attempting a port to 2.x would
be your best option.
hth
Lewis
[0]
http://svn.apache.org/repos/asf/nutch/trunk/src/java/org/apache/nutch/crawl/MimeAdaptiveFetchSchedule.java
[1] http://svn.apache.org/repos/asf/nutch/trunk/conf/adaptive-mimetypes.txt