You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by rk_sharma <rk...@yahoo.com> on 2013/12/18 22:40:17 UTC

Exception in NUTCH 2.2.1

Hi i am using nutch on rhel-5 and facing an exception 

[root@localhost local]# bin/nutch crawl urls -dir crawl -depth 3 -topN 5
InjectorJob: Using class org.apache.gora.memory.store.MemStore as the Gora
storage class.
Exception in thread "main"
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
not exist: file:/root/Desktop/apache-nutch-2.2.1/runtime/local/crawl
        at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
        at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
        at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
        at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
        at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
        at
org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:50)
        at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:233)
        at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
        at org.apache.nutch.crawl.Crawler.run(Crawler.java:136)
        at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)

but when i manually create crawl folder in ${NUTCH_HOME}/local then i get
following exception

[root@localhost local]# bin/nutch crawl urls -dir crawl -depth 3 -topN 5
InjectorJob: Using class org.apache.gora.memory.store.MemStore as the Gora
storage class.
InjectorJob: total number of urls rejected by filters: 0
InjectorJob: total number of urls injected after normalization and
filtering: 0
Exception in thread "main" java.lang.RuntimeException: job failed:
name=generate: null, jobid=job_local892109497_0002
        at
org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
        at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199)
        at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
        at org.apache.nutch.crawl.Crawler.run(Crawler.java:152)
        at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)
 
P.S. can anyone tell me what step should i follow to remove these exception.
Is there any link between previous exception on this exception



--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Exception in NUTCH 2.2.1

Posted by rk_sharma <rk...@yahoo.com>.
No it's just a convention to write APACHE_NUTCH_HOME. it shows your root
directory where you unzip your nutch setup files. 



--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326p4163161.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Exception in NUTCH 2.2.1

Posted by sagarhandore007 <sa...@gmail.com>.
Had you set APACHE_NUTCH_HOME variable to /home/.../apche-nutch-2.2.1/ ????



--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326p4163132.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Exception in NUTCH 2.2.1

Posted by rk_sharma <rk...@yahoo.com>.

Hi.

Thanks for your suggestion. 
I wants to use MySql as storage backend for Gora in Nutch-2.2.1 is their any proper documentation that can be followed. Last two days i am searching on that but could not find any documentation for this.

Thanks 
Rajni Kant Sharma
रजनी कान्त शर्मा



On Thursday, 19 December 2013 6:18 PM, lewis john mcgibbney [via Lucene] <ml...@n3.nabble.com> wrote:
 
Hi rk_sharma, 

On Wed, Dec 18, 2013 at 9:40 PM, <[hidden email]> wrote: 

> 
> Hi i am using nutch on rhel-5 and facing an exception 
> 
> [root@localhost local]# bin/nutch crawl urls -dir crawl -depth 3 -topN 5 
> InjectorJob: Using class org.apache.gora.memory.store.MemStore as the Gora 
> storage class. 
> Exception in thread "main" 
> 

[SNIP] 


> 
> P.S. can anyone tell me what step should i follow to remove these 
> exception. 
> Is there any link between previous exception on this exception 
> 
> I would strongly advice you to follow the Nutch2Tutorial. 
https://wiki.apache.org/nutch/Nutch2Tutorial
However I would also strongly advise that if you are using the MemStore 
then you most likely do NOT need to use Nutch 2.X for your tasks. 
I would consider using Nutch 1.X. There is a tutorial for the latter on the 
wiki as well. 
HTH 
Lewis 


________________________________
 
If you reply to this email, your message will be added to the discussion below:http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326p4107425.html 
To unsubscribe from Exception in NUTCH 2.2.1, click here.
NAML



--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326p4107427.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Exception in NUTCH 2.2.1

Posted by rk_sharma <rk...@yahoo.com>.
Thanks for your suggestion Now i have used http://nlp.solutions.asia/?p=180
link and there is no error but my database is blank. output is following

[root@localhost local]# bin/nutch crawl urls -depth 3 -topN 5
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : -1
Using queue mode : byHost
Fetcher: threads: 10
QueueFeeder finished: total 0 records. Hit by time limit :0
-finishing thread FetcherThread0, activeThreads=0
Fetcher: throughput threshold: -1
-finishing thread FetcherThread1, activeThreads=7
-finishing thread FetcherThread2, activeThreads=6
-finishing thread FetcherThread3, activeThreads=5
-finishing thread FetcherThread4, activeThreads=4
-finishing thread FetcherThread7, activeThreads=3
-finishing thread FetcherThread6, activeThreads=2
-finishing thread FetcherThread5, activeThreads=1
-finishing thread FetcherThread8, activeThreads=0
Fetcher: throughput threshold sequence: 5
-finishing thread FetcherThread9, activeThreads=0
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0.0 pages/s, 0 0 kb/s, 0 URLs
in 0 queues
-activeThreads=0
ParserJob: resuming:	false
ParserJob: forced reparse:	false
ParserJob: parsing all
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : -1
Using queue mode : byHost
Fetcher: threads: 10
QueueFeeder finished: total 0 records. Hit by time limit :0
-finishing thread FetcherThread0, activeThreads=0
-finishing thread FetcherThread6, activeThreads=5
-finishing thread FetcherThread1, activeThreads=4
-finishing thread FetcherThread2, activeThreads=3
-finishing thread FetcherThread3, activeThreads=2
-finishing thread FetcherThread4, activeThreads=1
-finishing thread FetcherThread5, activeThreads=0
-finishing thread FetcherThread7, activeThreads=0
Fetcher: throughput threshold: -1
Fetcher: throughput threshold sequence: 5
-finishing thread FetcherThread8, activeThreads=0
-finishing thread FetcherThread9, activeThreads=0
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0.0 pages/s, 0 0 kb/s, 0 URLs
in 0 queues
-activeThreads=0
ParserJob: resuming:	false
ParserJob: forced reparse:	false
ParserJob: parsing all
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : -1
Using queue mode : byHost
Fetcher: threads: 10
QueueFeeder finished: total 0 records. Hit by time limit :0
-finishing thread FetcherThread0, activeThreads=0
-finishing thread FetcherThread1, activeThreads=1
-finishing thread FetcherThread2, activeThreads=0
-finishing thread FetcherThread3, activeThreads=0
-finishing thread FetcherThread5, activeThreads=0
-finishing thread FetcherThread6, activeThreads=0
-finishing thread FetcherThread4, activeThreads=0
-finishing thread FetcherThread7, activeThreads=0
-finishing thread FetcherThread8, activeThreads=0
Fetcher: throughput threshold: -1
Fetcher: throughput threshold sequence: 5
-finishing thread FetcherThread9, activeThreads=0
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0.0 pages/s, 0 0 kb/s, 0 URLs
in 0 queues
-activeThreads=0
ParserJob: resuming:	false
ParserJob: forced reparse:	false
ParserJob: parsing all

my crawl url is 

http://www.spineor.com

and regex is 

+^http://([a-z0-9]*\.)*spineor.com/([a-z0-9]*\.)

what can be error.




--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-NUTCH-2-2-1-tp4107326p4107660.html
Sent from the Nutch - User mailing list archive at Nabble.com.