You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Donald MacDonald <do...@hotmail.com> on 2012/11/21 19:23:06 UTC
Run nutch 2.1 in distributed mode
Hi
I am trying to run nutch 2.1 in distributed mode. I have it working locally
saving to Hbase. When I try and set it up to run in distributed mode it
seems not to see my local settings. I get the error:
12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 12 more
Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
... 15 more
Which is funny as I am not using SQL DB.
I am using the command
$NUTCH_HOME/bin/nutch inject seed/urls
Where $NUTCH_HOME= /nutch2/runtime/deploy
I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much
success.
I am wondering what I am doing wrong.
Regards
D.
Re: Run nutch 2.1 in distributed mode
Posted by Donald MacDonald <do...@hotmail.com>.
Hi Lewis,
Ah, that makes sense, now you mention it :-)
Thanks
Donald.
On 21 Nov 2012, at 18:38, "Lewis John Mcgibbney" <le...@gmail.com> wrote:
> Hi Donald,
>
> I would advise you to re-generate the nutch job archive, as it appears
> that your settings are not included within the job file you are trying
> to deploy on your hadoop setup/cluster.
>
> you can do this by running ant job (after making changes to the files
> in conf) from $NUTCH_HOME
>
> hth
>
> Lewis
>
> On Wed, Nov 21, 2012 at 6:23 PM, Donald MacDonald
> <do...@hotmail.com> wrote:
>> Hi
>>
>> I am trying to run nutch 2.1 in distributed mode. I have it working locally
>> saving to Hbase. When I try and set it up to run in distributed mode it
>> seems not to see my local settings. I get the error:
>> 12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
>> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
>> at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
>> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
>> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
>> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>> at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
>> at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
>> at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
>> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
>> ... 12 more
>> Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:186)
>> at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
>> ... 15 more
>>
>>
>> Which is funny as I am not using SQL DB.
>>
>> I am using the command
>> $NUTCH_HOME/bin/nutch inject seed/urls
>>
>> Where $NUTCH_HOME= /nutch2/runtime/deploy
>>
>> I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much
>> success.
>>
>> I am wondering what I am doing wrong.
>>
>> Regards
>>
>> D.
>
>
>
> --
> Lewis
Re: Run nutch 2.1 in distributed mode
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Donald,
I would advise you to re-generate the nutch job archive, as it appears
that your settings are not included within the job file you are trying
to deploy on your hadoop setup/cluster.
you can do this by running ant job (after making changes to the files
in conf) from $NUTCH_HOME
hth
Lewis
On Wed, Nov 21, 2012 at 6:23 PM, Donald MacDonald
<do...@hotmail.com> wrote:
> Hi
>
> I am trying to run nutch 2.1 in distributed mode. I have it working locally
> saving to Hbase. When I try and set it up to run in distributed mode it
> seems not to see my local settings. I get the error:
> 12/11/21 17:50:58 ERROR crawl.InjectorJob: InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
> at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:75)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:214)
> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:228)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:248)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:258)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
> at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:747)
> at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:160)
> at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 12 more
> Caused by: java.lang.ClassNotFoundException: org.hsqldb.jdbc.JDBCDriver
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:186)
> at org.apache.gora.sql.store.SqlStore.getConnection(SqlStore.java:735)
> ... 15 more
>
>
> Which is funny as I am not using SQL DB.
>
> I am using the command
> $NUTCH_HOME/bin/nutch inject seed/urls
>
> Where $NUTCH_HOME= /nutch2/runtime/deploy
>
> I have searched for a reference/tutorial on setting nutch 2.0 up on Hadoop with not much
> success.
>
> I am wondering what I am doing wrong.
>
> Regards
>
> D.
>
>
--
Lewis