processing parallel sites

[bruce <be...@earthlink.net>]

hi...

can nutch/lucene handle getting content from 100s-1000s of pages
sinultaneously....

if it can, how does it write the content to the resulting/output db.

does it actually perform 100s-1000s of simultaneous connections to a backend
db.

does it utilize writing the output files to a filesystem, which is then
somehow inserted into a db....

thanks

-bruce

Re: indexing folders with nutch

[Lourival Júnior <ju...@gmail.com>]

Yes Cam, if you use a depth 1 you will crawl only the first document. With a
depth 2 you will crawl the first document and all the links found on this
document. With depth 3, you will crawl the first one, its links and all
links found in cycle 2. And so on. Increasing you depth will increasing your
WebDB too. Try it ;)

Regards

On 8/31/06, Sandy Polanski <sa...@yahoo.com> wrote:
>
> Cam, try increasing the depth and see what happens.
> It seems that logic would say that they're on the same
> directory depth/level; however, just give it a try
> because I ran into a similar problem, and if I'm not
> mistaken, that fixed it.
>
> --- Cam Bazz <ca...@gmail.com> wrote:
>
> > Hello,
> >
> > I have a problem. I tried to index some localfiles
> > with nutch.
> >
> > What I have done is put them in a local apache
> > server, (html files)
> > and create a urls file that contains
> > http://localhost/file01.html etc.
> >
> > then I do a nutch crawl urls . -dir crawl -depth 1
> >
> > but the crawl stales after a while, and nothing
> > happens.
> >
> > I also tried -topN 10000
> >
> > is not there a more convinient way of indexing from
> > file system?
> >
> > Best regards,
> > -C.B.
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>



-- 
Lourival Junior
Universidade Federal do Pará
Curso de Bacharelado em Sistemas de Informação
http://www.ufpa.br/cbsi
Msn: junior_ufpa@hotmail.com

Re: indexing folders with nutch

[Sandy Polanski <sa...@yahoo.com>]

Cam, try increasing the depth and see what happens. 
It seems that logic would say that they're on the same
directory depth/level; however, just give it a try
because I ran into a similar problem, and if I'm not
mistaken, that fixed it.

--- Cam Bazz <ca...@gmail.com> wrote:

> Hello,
> 
> I have a problem. I tried to index some localfiles
> with nutch.
> 
> What I have done is put them in a local apache
> server, (html files)
> and create a urls file that contains
> http://localhost/file01.html etc.
> 
> then I do a nutch crawl urls . -dir crawl -depth 1
> 
> but the crawl stales after a while, and nothing
> happens.
> 
> I also tried -topN 10000
> 
> is not there a more convinient way of indexing from
> file system?
> 
> Best regards,
> -C.B.
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

indexing folders with nutch

[Cam Bazz <ca...@gmail.com>]

Hello,

I have a problem. I tried to index some localfiles with nutch.

What I have done is put them in a local apache server, (html files)
and create a urls file that contains http://localhost/file01.html etc.

then I do a nutch crawl urls . -dir crawl -depth 1

but the crawl stales after a while, and nothing happens.

I also tried -topN 10000

is not there a more convinient way of indexing from file system?

Best regards,
-C.B.

RE: processing parallel sites

[bruce <be...@earthlink.net>]

Hi Ken!!

Thanks for the short conversation regarding the project. You've more or less
confirmed some of the initial thoughts/research that we've observed.

Regards,

-Bruce Douglas
bedouglas@earthlink.net



-----Original Message-----
From: Ken Krugler [mailto:kkrugler_lists@transpac.com]
Sent: Monday, August 28, 2006 12:31 PM
To: nutch-user@lucene.apache.org
Subject: Re: processing parallel sites


>can nutch/lucene handle getting content from 100s-1000s of pages
>sinultaneously....

If you mean "Can Nutch handle fetching 1000s of pages at a time", the
answer is yes.

If you mean "Can Lucene, when used as the IR engine for Nutch, handle
searching 1000s of pages at a time", then answer is also yes.

>if it can, how does it write the content to the resulting/output db.
>
>does it actually perform 100s-1000s of simultaneous connections to a
backend
>db.
>
>does it utilize writing the output files to a filesystem, which is then
>somehow inserted into a db....

The fetched pages get written to a sequential file. After a fetch
cycle, additional data about the page state gets processed, and the
results are used to update the crawldb, which is a kind of
specialized database for web crawling.

-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"

Re: processing parallel sites

[Ken Krugler <kk...@transpac.com>]

>can nutch/lucene handle getting content from 100s-1000s of pages
>sinultaneously....

If you mean "Can Nutch handle fetching 1000s of pages at a time", the 
answer is yes.

If you mean "Can Lucene, when used as the IR engine for Nutch, handle 
searching 1000s of pages at a time", then answer is also yes.

>if it can, how does it write the content to the resulting/output db.
>
>does it actually perform 100s-1000s of simultaneous connections to a backend
>db.
>
>does it utilize writing the output files to a filesystem, which is then
>somehow inserted into a db....

The fetched pages get written to a sequential file. After a fetch 
cycle, additional data about the page state gets processed, and the 
results are used to update the crawldb, which is a kind of 
specialized database for web crawling.

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"