You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by AJ Chen <ca...@gmail.com> on 2005/09/13 22:20:40 UTC
how to reuse webDB with new urls
Once I create a webDB, can I inject new root urls to the same webDB
repeatly? After each injection, run as many cycles of
generate/fetch/updatedb to fetch all web pages from the new sites. I think
this will allow me to gradually build a comprehensive vertical site. Any
comment or suggestion?
-AJ
Re: how to reuse webDB with new urls
Posted by AJ Chen <an...@sbcglobal.net>.
Before re-injecting a new set of urls to webdb, I'll wait until all
fetch operations (generate + fetch + updatedb) are done. I'm not sure
it's necessary or not, but it's cleaner.
One more question: Should I run UpdateSegmentsFromDb to update the
segments before any new injection? Does segment updating affect url
injection and fetch list generation?
-AJ
Jay Lorenzo wrote:
>What about the issue of maintaining some semblance of ACIDity? Don't you
>have to make sure that the generation of fetchlists and the updates are run
>synchronously, ie one update or generate at a time?
>
>On 9/13/05, Michael Ji <fj...@yahoo.com> wrote:
>
>
>>I think this scenario will work.
>>
>>Just a bit worry about the filter performance if the
>>domain site number is in scale of thundreds of
>>thousands.
>>
>>Michael Ji
>>
>>--- AJ Chen <ca...@gmail.com> wrote:
>>
>>
>>
>>>Once I create a webDB, can I inject new root urls to
>>>the same webDB
>>>repeatly? After each injection, run as many cycles
>>>of
>>>generate/fetch/updatedb to fetch all web pages from
>>>the new sites. I think
>>>this will allow me to gradually build a
>>>comprehensive vertical site. Any
>>>comment or suggestion?
>>>-AJ
>>>
>>>
>>>
>>
>>
>>__________________________________
>>Yahoo! Mail - PC Magazine Editors' Choice 2005
>>http://mail.yahoo.com
>>
>>
>>
>
>
>
--
AJ (Anjun) Chen, Ph.D.
Canova Bioconsulting
Marketing * BD * Software Development
748 Matadero Ave., Palo Alto, CA 94306, USA
Cell 650-283-4091, anjun.chen@sbcglobal.net
---------------------------------------------------
Re: how to reuse webDB with new urls
Posted by Jay Lorenzo <ja...@gmail.com>.
What about the issue of maintaining some semblance of ACIDity? Don't you
have to make sure that the generation of fetchlists and the updates are run
synchronously, ie one update or generate at a time?
On 9/13/05, Michael Ji <fj...@yahoo.com> wrote:
>
> I think this scenario will work.
>
> Just a bit worry about the filter performance if the
> domain site number is in scale of thundreds of
> thousands.
>
> Michael Ji
>
> --- AJ Chen <ca...@gmail.com> wrote:
>
> > Once I create a webDB, can I inject new root urls to
> > the same webDB
> > repeatly? After each injection, run as many cycles
> > of
> > generate/fetch/updatedb to fetch all web pages from
> > the new sites. I think
> > this will allow me to gradually build a
> > comprehensive vertical site. Any
> > comment or suggestion?
> > -AJ
> >
>
>
>
>
> __________________________________
> Yahoo! Mail - PC Magazine Editors' Choice 2005
> http://mail.yahoo.com
>
Re: how to reuse webDB with new urls
Posted by Michael Ji <fj...@yahoo.com>.
I think this scenario will work.
Just a bit worry about the filter performance if the
domain site number is in scale of thundreds of
thousands.
Michael Ji
--- AJ Chen <ca...@gmail.com> wrote:
> Once I create a webDB, can I inject new root urls to
> the same webDB
> repeatly? After each injection, run as many cycles
> of
> generate/fetch/updatedb to fetch all web pages from
> the new sites. I think
> this will allow me to gradually build a
> comprehensive vertical site. Any
> comment or suggestion?
> -AJ
>
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com