You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Nahuel ANGELINETTI <na...@develog.com> on 2006/08/03 11:50:57 UTC
Recrawl urls
Hello,
I was searching for the method to add new url to the crawling url list
and how to recrawl all urls...
Can you help me ?
thanks,
--
Nahuel ANGELINETTI
Re: Recrawl urls
Posted by Nahuel ANGELINETTI <na...@develog.com>.
Hi,
Thanks a lot of those informationations, their are very helpful.
Happy to see the community is active !
Regards,
--
Nahuel ANGELINETTI
Le Thu, 3 Aug 2006 08:31:22 -0300,
"Lourival Júnior" <ju...@gmail.com> a écrit :
> Hi Nahuel!
>
> You could use the command bin/nutch inject $nutch-dir/db -urlfile
> urlfile.txt. To recrawl your WebDB you can use this
> script.<http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
>
> Take a look to the adddays argument and to the configuration property
> db.default.fetch.interval.They influence to the result.
>
> Regards!
>
> On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> >
> > Hello,
> >
> > I was searching for the method to add new url to the crawling url
> > list and how to recrawl all urls...
> >
> > Can you help me ?
> >
> > thanks,
> >
> > --
> > Nahuel ANGELINETTI
> >
>
>
>
Re: Recrawl urls
Posted by aidahaj <ai...@gmail.com>.
Thanks for these information about recrawling.
I am running a recrawling operation but every time I do it, I don't get the
same results as the first crawl(different documents , not the same web
pages). So how can I handle to recrawl same pages?
Maybe fixe the property db.default.fetch.interval to 0 or maybe this will
generate an error?
--
View this message in context: http://www.nabble.com/Recrawl-urls-tp5629689p23543078.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Recrawl urls
Posted by Lourival Júnior <ju...@gmail.com>.
This command "bin/nutch purge" doesn't exist. Well I can't say you what is
happening. Give me the output when you run the recrawl.
On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
>
> 0.7.2 of nutch
>
> Le Thu, 3 Aug 2006 09:37:24 -0300,
> "Lourival Júnior" <ju...@gmail.com> a écrit :
>
> > Which version are you using?
> >
> > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > >
> > > But the websites just added hasn't been yet crawled... And they're
> > > not crawled during recrawl...
> > > Does "bin/nutch purge" will restart all ?
> > >
> > >
> > >
> > > Le Thu, 3 Aug 2006 09:21:04 -0300,
> > > "Lourival Júnior" <ju...@gmail.com> a écrit :
> > >
> > > > In the nutch conf/nutch-default.xml configuration file exist a
> > > > property call db.default.fetch.interval. When you crawl a site,
> > > > nutch schedules the next fetch to "today +
> > > > db.default.fetch.interval" days. If execute the recrawl command
> > > > and the pages that you fetch don't reach this date, they won't be
> > > > re-fetched. When you add new urls to the webdb, they will be
> > > > ready to be fetch. So at this moment only this pages will be
> > > > fetched by the recrawl script.
> > > >
> > > > I hope I helped you. If I said some wrong thing, please correct
> > > > me :)
> > > >
> > > > Regards
> > > >
> > > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > > >
> > > > > I have another question, I done what you give me... But it
> > > > > inject the new urls and "recrawl" it, but against the first
> > > > > crawl It doesn't download the web pages and really crawl
> > > > > them... perhaps I'm mistaking somewhere...
> > > > > Any idea ?
> > > > >
> > > > > Regards,
> > > > >
> > > > > --
> > > > > Nahuel ANGELINETTI
> > > > >
> > > > > Le Thu, 3 Aug 2006 08:31:22 -0300,
> > > > > "Lourival Júnior" <ju...@gmail.com> a écrit :
> > > > >
> > > > > > Hi Nahuel!
> > > > > >
> > > > > > You could use the command bin/nutch inject $nutch-dir/db
> > > > > > -urlfile urlfile.txt. To recrawl your WebDB you can use this
> > > > > > script.<
> > > > >
> > >
> http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
> > > > > >
> > > > > > Take a look to the adddays argument and to the configuration
> > > > > > property db.default.fetch.interval.They influence to the
> > > > > > result.
> > > > > >
> > > > > > Regards!
> > > > > >
> > > > > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I was searching for the method to add new url to the
> > > > > > > crawling url list and how to recrawl all urls...
> > > > > > >
> > > > > > > Can you help me ?
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Nahuel ANGELINETTI
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
--
Lourival Junior
Universidade Federal do Pará
Curso de Bacharelado em Sistemas de Informação
http://www.ufpa.br/cbsi
Msn: junior_ufpa@hotmail.com
Re: Recrawl urls
Posted by Nahuel ANGELINETTI <na...@develog.com>.
0.7.2 of nutch
Le Thu, 3 Aug 2006 09:37:24 -0300,
"Lourival Júnior" <ju...@gmail.com> a écrit :
> Which version are you using?
>
> On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> >
> > But the websites just added hasn't been yet crawled... And they're
> > not crawled during recrawl...
> > Does "bin/nutch purge" will restart all ?
> >
> >
> >
> > Le Thu, 3 Aug 2006 09:21:04 -0300,
> > "Lourival Júnior" <ju...@gmail.com> a écrit :
> >
> > > In the nutch conf/nutch-default.xml configuration file exist a
> > > property call db.default.fetch.interval. When you crawl a site,
> > > nutch schedules the next fetch to "today +
> > > db.default.fetch.interval" days. If execute the recrawl command
> > > and the pages that you fetch don't reach this date, they won't be
> > > re-fetched. When you add new urls to the webdb, they will be
> > > ready to be fetch. So at this moment only this pages will be
> > > fetched by the recrawl script.
> > >
> > > I hope I helped you. If I said some wrong thing, please correct
> > > me :)
> > >
> > > Regards
> > >
> > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > >
> > > > I have another question, I done what you give me... But it
> > > > inject the new urls and "recrawl" it, but against the first
> > > > crawl It doesn't download the web pages and really crawl
> > > > them... perhaps I'm mistaking somewhere...
> > > > Any idea ?
> > > >
> > > > Regards,
> > > >
> > > > --
> > > > Nahuel ANGELINETTI
> > > >
> > > > Le Thu, 3 Aug 2006 08:31:22 -0300,
> > > > "Lourival Júnior" <ju...@gmail.com> a écrit :
> > > >
> > > > > Hi Nahuel!
> > > > >
> > > > > You could use the command bin/nutch inject $nutch-dir/db
> > > > > -urlfile urlfile.txt. To recrawl your WebDB you can use this
> > > > > script.<
> > > >
> > http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
> > > > >
> > > > > Take a look to the adddays argument and to the configuration
> > > > > property db.default.fetch.interval.They influence to the
> > > > > result.
> > > > >
> > > > > Regards!
> > > > >
> > > > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I was searching for the method to add new url to the
> > > > > > crawling url list and how to recrawl all urls...
> > > > > >
> > > > > > Can you help me ?
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > --
> > > > > > Nahuel ANGELINETTI
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
Re: Recrawl urls
Posted by Lourival Júnior <ju...@gmail.com>.
Which version are you using?
On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
>
> But the websites just added hasn't been yet crawled... And they're not
> crawled during recrawl...
> Does "bin/nutch purge" will restart all ?
>
>
>
> Le Thu, 3 Aug 2006 09:21:04 -0300,
> "Lourival Júnior" <ju...@gmail.com> a écrit :
>
> > In the nutch conf/nutch-default.xml configuration file exist a
> > property call db.default.fetch.interval. When you crawl a site, nutch
> > schedules the next fetch to "today + db.default.fetch.interval" days.
> > If execute the recrawl command and the pages that you fetch don't
> > reach this date, they won't be re-fetched. When you add new urls to
> > the webdb, they will be ready to be fetch. So at this moment only
> > this pages will be fetched by the recrawl script.
> >
> > I hope I helped you. If I said some wrong thing, please correct me :)
> >
> > Regards
> >
> > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > >
> > > I have another question, I done what you give me... But it inject
> > > the new urls and "recrawl" it, but against the first crawl It
> > > doesn't download the web pages and really crawl them... perhaps I'm
> > > mistaking somewhere...
> > > Any idea ?
> > >
> > > Regards,
> > >
> > > --
> > > Nahuel ANGELINETTI
> > >
> > > Le Thu, 3 Aug 2006 08:31:22 -0300,
> > > "Lourival Júnior" <ju...@gmail.com> a écrit :
> > >
> > > > Hi Nahuel!
> > > >
> > > > You could use the command bin/nutch inject $nutch-dir/db -urlfile
> > > > urlfile.txt. To recrawl your WebDB you can use this
> > > > script.<
> > >
> http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
> > > >
> > > > Take a look to the adddays argument and to the configuration
> > > > property db.default.fetch.interval.They influence to the result.
> > > >
> > > > Regards!
> > > >
> > > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I was searching for the method to add new url to the crawling
> > > > > url list and how to recrawl all urls...
> > > > >
> > > > > Can you help me ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > --
> > > > > Nahuel ANGELINETTI
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
--
Lourival Junior
Universidade Federal do Pará
Curso de Bacharelado em Sistemas de Informação
http://www.ufpa.br/cbsi
Msn: junior_ufpa@hotmail.com
Re: Recrawl urls
Posted by Nahuel ANGELINETTI <na...@develog.com>.
But the websites just added hasn't been yet crawled... And they're not
crawled during recrawl...
Does "bin/nutch purge" will restart all ?
Le Thu, 3 Aug 2006 09:21:04 -0300,
"Lourival Júnior" <ju...@gmail.com> a écrit :
> In the nutch conf/nutch-default.xml configuration file exist a
> property call db.default.fetch.interval. When you crawl a site, nutch
> schedules the next fetch to "today + db.default.fetch.interval" days.
> If execute the recrawl command and the pages that you fetch don't
> reach this date, they won't be re-fetched. When you add new urls to
> the webdb, they will be ready to be fetch. So at this moment only
> this pages will be fetched by the recrawl script.
>
> I hope I helped you. If I said some wrong thing, please correct me :)
>
> Regards
>
> On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> >
> > I have another question, I done what you give me... But it inject
> > the new urls and "recrawl" it, but against the first crawl It
> > doesn't download the web pages and really crawl them... perhaps I'm
> > mistaking somewhere...
> > Any idea ?
> >
> > Regards,
> >
> > --
> > Nahuel ANGELINETTI
> >
> > Le Thu, 3 Aug 2006 08:31:22 -0300,
> > "Lourival Júnior" <ju...@gmail.com> a écrit :
> >
> > > Hi Nahuel!
> > >
> > > You could use the command bin/nutch inject $nutch-dir/db -urlfile
> > > urlfile.txt. To recrawl your WebDB you can use this
> > > script.<
> > http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
> > >
> > > Take a look to the adddays argument and to the configuration
> > > property db.default.fetch.interval.They influence to the result.
> > >
> > > Regards!
> > >
> > > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I was searching for the method to add new url to the crawling
> > > > url list and how to recrawl all urls...
> > > >
> > > > Can you help me ?
> > > >
> > > > thanks,
> > > >
> > > > --
> > > > Nahuel ANGELINETTI
> > > >
> > >
> > >
> > >
> >
>
>
>
Re: Recrawl urls
Posted by Lourival Júnior <ju...@gmail.com>.
In the nutch conf/nutch-default.xml configuration file exist a property call
db.default.fetch.interval. When you crawl a site, nutch schedules the next
fetch to "today + db.default.fetch.interval" days. If execute the recrawl
command and the pages that you fetch don't reach this date, they won't be
re-fetched. When you add new urls to the webdb, they will be ready to be
fetch. So at this moment only this pages will be fetched by the recrawl
script.
I hope I helped you. If I said some wrong thing, please correct me :)
Regards
On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
>
> I have another question, I done what you give me... But it inject the
> new urls and "recrawl" it, but against the first crawl It doesn't
> download the web pages and really crawl them... perhaps I'm mistaking
> somewhere...
> Any idea ?
>
> Regards,
>
> --
> Nahuel ANGELINETTI
>
> Le Thu, 3 Aug 2006 08:31:22 -0300,
> "Lourival Júnior" <ju...@gmail.com> a écrit :
>
> > Hi Nahuel!
> >
> > You could use the command bin/nutch inject $nutch-dir/db -urlfile
> > urlfile.txt. To recrawl your WebDB you can use this
> > script.<
> http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
> >
> > Take a look to the adddays argument and to the configuration property
> > db.default.fetch.interval.They influence to the result.
> >
> > Regards!
> >
> > On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> > >
> > > Hello,
> > >
> > > I was searching for the method to add new url to the crawling url
> > > list and how to recrawl all urls...
> > >
> > > Can you help me ?
> > >
> > > thanks,
> > >
> > > --
> > > Nahuel ANGELINETTI
> > >
> >
> >
> >
>
--
Lourival Junior
Universidade Federal do Pará
Curso de Bacharelado em Sistemas de Informação
http://www.ufpa.br/cbsi
Msn: junior_ufpa@hotmail.com
Re: Recrawl urls
Posted by Nahuel ANGELINETTI <na...@develog.com>.
I have another question, I done what you give me... But it inject the
new urls and "recrawl" it, but against the first crawl It doesn't
download the web pages and really crawl them... perhaps I'm mistaking
somewhere...
Any idea ?
Regards,
--
Nahuel ANGELINETTI
Le Thu, 3 Aug 2006 08:31:22 -0300,
"Lourival Júnior" <ju...@gmail.com> a écrit :
> Hi Nahuel!
>
> You could use the command bin/nutch inject $nutch-dir/db -urlfile
> urlfile.txt. To recrawl your WebDB you can use this
> script.<http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
>
> Take a look to the adddays argument and to the configuration property
> db.default.fetch.interval.They influence to the result.
>
> Regards!
>
> On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
> >
> > Hello,
> >
> > I was searching for the method to add new url to the crawling url
> > list and how to recrawl all urls...
> >
> > Can you help me ?
> >
> > thanks,
> >
> > --
> > Nahuel ANGELINETTI
> >
>
>
>
Re: Recrawl urls
Posted by Lourival Júnior <ju...@gmail.com>.
Hi Nahuel!
You could use the command bin/nutch inject $nutch-dir/db -urlfile
urlfile.txt. To recrawl your WebDB you can use this
script.<http://today.java.net/pub/a/today/2006/02/16/introduction-to-nutch-2.html>
Take a look to the adddays argument and to the configuration property
db.default.fetch.interval.They influence to the result.
Regards!
On 8/3/06, Nahuel ANGELINETTI <na...@develog.com> wrote:
>
> Hello,
>
> I was searching for the method to add new url to the crawling url list
> and how to recrawl all urls...
>
> Can you help me ?
>
> thanks,
>
> --
> Nahuel ANGELINETTI
>
--
Lourival Junior
Universidade Federal do Pará
Curso de Bacharelado em Sistemas de Informação
http://www.ufpa.br/cbsi
Msn: junior_ufpa@hotmail.com