my own crawlscript.sh

["Matthias W." <Ma...@e-projecta.com>]

Hi,
I've got a textfile with all URLs to index, I don't want to crawl URLs
before indexing.
How to do this?

Also I'm creating an index in a temporary folder and on success I want to
overwrite the old index.
How do I check in the shell script, if the crawl- (index-) command was
successful?
-- 
View this message in context: http://www.nabble.com/my-own-crawlscript.sh-tp20853413p20853413.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: my own crawlscript.sh

["Matthias W." <Ma...@e-projecta.com>]

Dennis Kubes-2 wrote:
> 
> Just having the urls isn't the same as having an index.  You would still 
> need to crawl them.  You can inject your url list into a clean crawldb 
> and fetch only those urls with the inject, generate, fetch commands. 
> Then you can use the index command to index them.
> 
OK, thanks. 

Dennis Kubes-2 wrote:
> 
> You could check size.  You could also check it programatically through 
> lucene.
> 
> Dennis
> 
You mean that I should check the size of my crawl-folder and its elements?
But this would be very buggy? What if an error occures during the crawl and
the crawl has already the required size? Or is this impossible?

And how can I check this programmatically?
-- 
View this message in context: http://www.nabble.com/my-own-crawlscript.sh-tp20853413p20894745.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: my own crawlscript.sh

[Dennis Kubes <ku...@apache.org>]

Matthias W. wrote:
> Hi,
> I've got a textfile with all URLs to index, I don't want to crawl URLs
> before indexing.

Just having the urls isn't the same as having an index.  You would still 
need to crawl them.  You can inject your url list into a clean crawldb 
and fetch only those urls with the inject, generate, fetch commands. 
Then you can use the index command to index them.

> How to do this?
> 
> Also I'm creating an index in a temporary folder and on success I want to
> overwrite the old index.
> How do I check in the shell script, if the crawl- (index-) command was
> successful?

You could check size.  You could also check it programatically through 
lucene.

Dennis