You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by CdnGuy <gu...@gmail.com> on 2014/07/04 20:40:57 UTC
NutchTutorial Followed Crawldb Not Created
Hi,
I've followed the NutchTutorial at
http://wiki.apache.org/nutch/NutchTutorial
<http://wiki.apache.org/nutch/NutchTutorial> to the letter twice.
The first time, the crawl worked correctly.
This time, the crawl happened but no crawldb linkdb or segments files were
created.
My first attempt has already been wiped so I can't compare configuration to
see if I made a mistake.
I am not getting any error messages.
I've reviewed my configuration and everything appears to be in order.
Here's what happens when I run the crawl...
root@myserver:~/nutch# bin/crawl urls/seed.txt testcrawl -dir crawl -depth 3
-topN 50
Injector: starting at 2014-07-04 14:27:57
Injector: crawlDb: testcrawl/crawldb
Injector: urlDir: urls/seed.txt
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
Injector: Merging injected urls into crawl db.
Injector: overwrite: false
Injector: update: false
Injector: finished at 2014-07-04 14:28:05, elapsed: 00:00:07
Not looking for someone to fix it, just point me in the right direction to
figure this out. I really want to learn Nutch/Solr inside and out. TIA!
--
View this message in context: http://lucene.472066.n3.nabble.com/NutchTutorial-Followed-Crawldb-Not-Created-tp4145668.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: NutchTutorial Followed Crawldb Not Created
Posted by CdnGuy <gu...@gmail.com>.
Thank you. That points me in the right direction. Sorry for the late reply.
Out sick as of late. Looking forward to getting back to working on this.
--
View this message in context: http://lucene.472066.n3.nabble.com/NutchTutorial-Followed-Crawldb-Not-Created-tp4145668p4147285.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: NutchTutorial Followed Crawldb Not Created
Posted by Sebastian Nagel <wa...@googlemail.com>.
The second call of bin/crawl seems to have swapped seed and crawl directory:
> root@Walleye:~/nutch# bin/crawl crawl urls -dir crawl -depth 3 -topN 5
The first call
root@myserver:~/nutch# bin/crawl urls/seed.txt testcrawl -dir crawl -depth 3 -topN 50
seems to have the order correct. But does also not follow entirely the scheme
% bin/crawl <seedDir> <crawlDir> <solrURL> <numberOfRounds>
The status message of Injector also indicates that the seed directory does not
contain URLs:
(first call)
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
=> one URL injected
(second call)
> Injector: total number of urls rejected by filters: 0
> Injector: total number of urls injected after normalization and filtering: 0
=> zero URLs
Sebastian
On 07/05/2014 02:57 AM, CdnGuy wrote:
> Wiped and rebuilt the server from scratch.
> Followed tutorial again.
> Here's the results:
> root@Walleye:~/nutch# bin/crawl crawl urls -dir crawl -depth 3 -topN 5
> Injector: starting at 2014-07-04 20:50:27
> Injector: crawlDb: urls/crawldb
> Injector: urlDir: crawl
> Injector: Converting injected urls to crawl db entries.
> Injector: total number of urls rejected by filters: 0
> Injector: total number of urls injected after normalization and filtering: 0
> Injector: Merging injected urls into crawl db.
> Injector: overwrite: false
> Injector: update: false
> Injector: finished at 2014-07-04 20:50:31, elapsed: 00:00:03
>
> Still no files in the crawl directory.
> What am I missing?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/NutchTutorial-Followed-Crawldb-Not-Created-tp4145668p4145686.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
Re: NutchTutorial Followed Crawldb Not Created
Posted by CdnGuy <gu...@gmail.com>.
Wiped and rebuilt the server from scratch.
Followed tutorial again.
Here's the results:
root@Walleye:~/nutch# bin/crawl crawl urls -dir crawl -depth 3 -topN 5
Injector: starting at 2014-07-04 20:50:27
Injector: crawlDb: urls/crawldb
Injector: urlDir: crawl
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 0
Injector: Merging injected urls into crawl db.
Injector: overwrite: false
Injector: update: false
Injector: finished at 2014-07-04 20:50:31, elapsed: 00:00:03
Still no files in the crawl directory.
What am I missing?
--
View this message in context: http://lucene.472066.n3.nabble.com/NutchTutorial-Followed-Crawldb-Not-Created-tp4145668p4145686.html
Sent from the Nutch - User mailing list archive at Nabble.com.