You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Donald MacDonald <do...@hotmail.com> on 2012/11/21 19:23:06 UTC

Run nutch 2.1 in distributed mode

Hi 

I am trying to run nutch 2.1 in distributed mode. I have it working locally 
saving to Hbase. When I try and set it up to run in distributed mode it 
seems not to see my local settings. I get the error:
12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
	at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
	at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
	at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
	at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
	at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
	at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
	at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
	at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
	at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
	at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
	... 12 more
Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
	at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:186)
	at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
	... 15 more


Which is funny as I am not using SQL DB. 

I am using the command 
$NUTCH_HOME/bin/nutch inject seed/urls 

Where $NUTCH_HOME= /nutch2/runtime/deploy

I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much 
success. 

I am wondering what I am doing wrong.

Regards

D.



Re: Run nutch 2.1 in distributed mode

Posted by Donald MacDonald <do...@hotmail.com>.
Hi Lewis,

Ah, that makes sense, now you mention it :-)

Thanks 

Donald.

On 21 Nov 2012, at 18:38, "Lewis John Mcgibbney" <le...@gmail.com> wrote:

> Hi Donald,
> 
> I would advise you to re-generate the nutch job archive, as it appears
> that your settings are not included within the job file you are trying
> to deploy on your hadoop setup/cluster.
> 
> you can do this by running ant job (after making changes to the files
> in conf) from $NUTCH_HOME
> 
> hth
> 
> Lewis
> 
> On Wed, Nov 21, 2012 at 6:23 PM, Donald MacDonald
> <do...@hotmail.com> wrote:
>> Hi
>> 
>> I am trying to run nutch 2.1 in distributed mode. I have it working locally
>> saving to Hbase. When I try and set it up to run in distributed mode it
>> seems not to see my local settings. I get the error:
>> 12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>>        at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
>>        at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
>>        at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
>>        at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
>>        at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
>>        at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>        at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:616)
>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>>        at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
>>        at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
>>        at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
>>        at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
>>        ... 12 more
>> Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:186)
>>        at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
>>        ... 15 more
>> 
>> 
>> Which is funny as I am not using SQL DB.
>> 
>> I am using the command
>> $NUTCH_HOME/bin/nutch inject seed/urls
>> 
>> Where $NUTCH_HOME= /nutch2/runtime/deploy
>> 
>> I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much
>> success.
>> 
>> I am wondering what I am doing wrong.
>> 
>> Regards
>> 
>> D.
> 
> 
> 
> -- 
> Lewis

Re: Run nutch 2.1 in distributed mode

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Donald,

I would advise you to re-generate the nutch job archive, as it appears
that your settings are not included within the job file you are trying
to deploy on your hadoop setup/cluster.

you can do this by running ant job (after making changes to the files
in conf) from $NUTCH_HOME

hth

Lewis

On Wed, Nov 21, 2012 at 6:23 PM, Donald MacDonald
<do...@hotmail.com> wrote:
> Hi
>
> I am trying to run nutch 2.1 in distributed mode. I have it working locally
> saving to Hbase. When I try and set it up to run in distributed mode it
> seems not to see my local settings. I get the error:
> 12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>         at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
>         at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
>         at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
>         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
>         at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
>         at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>         at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
>         at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
>         at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
>         at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
>         ... 12 more
> Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:186)
>         at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
>         ... 15 more
>
>
> Which is funny as I am not using SQL DB.
>
> I am using the command
> $NUTCH_HOME/bin/nutch inject seed/urls
>
> Where $NUTCH_HOME= /nutch2/runtime/deploy
>
> I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much
> success.
>
> I am wondering what I am doing wrong.
>
> Regards
>
> D.
>
>



-- 
Lewis