You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Ryan L. Sun" <li...@gmail.com> on 2012/08/13 19:11:25 UTC
nutch 2.0 with hbase 0.94.0
hi all,
I'm trying to set up nutch 2.0 with a existing hbase cluster (using
hbase 0.94.0). Since nutch 2.0 supports an older version (0.90.4) of
hbase, starting a nutch inject job crashed hbase daemon. Copying hbase
0.94.0's lib to nutch/runtime/local/lib folder as google search hinted
doesn't work for me.
Any suggestions are appreciated. Thanks.
PS. I couldn't downgrade the existing hbase cluster software version,
which is out of my hand.
Re: nutch 2.0 with hbase 0.94.0
Posted by al...@aim.com.
did you delete the old hbase jar from the lib dir?
Alex.
-----Original Message-----
From: Lewis John Mcgibbney <le...@gmail.com>
To: user <us...@nutch.apache.org>
Sent: Mon, Aug 13, 2012 10:16 am
Subject: Re: nutch 2.0 with hbase 0.94.0
Nutch contains no knowledge of which specific version of a backend you
are using. This is however done through the gora-* dependencies
managed by Ivy.
Although this is a pretty convoluted way to do things, the best way to
find this would be to check out Gora trunk [0], upgrade the hbase
dependencies to whatever you need, compile and package the project
then copy the relevant jar's over to your Nutch installation. This way
you could run a standalone (development) hbase server and try running
your Nutch configuration that way...
hth
Lewis
[0] http://svn.apache.org/repos/asf/gora/trunk/
On Mon, Aug 13, 2012 at 6:11 PM, Ryan L. Sun <li...@gmail.com> wrote:
> hi all,
>
> I'm trying to set up nutch 2.0 with a existing hbase cluster (using
> hbase 0.94.0). Since nutch 2.0 supports an older version (0.90.4) of
> hbase, starting a nutch inject job crashed hbase daemon. Copying hbase
> 0.94.0's lib to nutch/runtime/local/lib folder as google search hinted
> doesn't work for me.
> Any suggestions are appreciated. Thanks.
>
> PS. I couldn't downgrade the existing hbase cluster software version,
> which is out of my hand.
--
Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Ryan,
Further to this I've raised this conversation over on user@gora[0] so
it might be of interest for you to keep up with it over there?
What is required is a determination of interoperability over HBase
versions and what Gora is currently capable of working with before we
require an API upgrade/change.
Best
Lewis
[0] http://www.mail-archive.com/user%40gora.apache.org/msg00048.html
On Mon, Aug 13, 2012 at 9:36 PM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> Hi Ryan,
>
> On Mon, Aug 13, 2012 at 7:36 PM, Ryan L. Sun <li...@gmail.com> wrote:
>> Nutch 2.0 is pre-configed with gora 0.2, and I checked gora website,
>> 0.2.1 is the latest release.
>> I edited ivy/ivy.xml to update gora-core/ gora-hbase version from 0.2
>> to 0.2.1 and then re-compile. It still fetched hbase-0.90.4.jar
>>
>
> yes the HBase jar is a transitive dependency which is pulled through
> gora-hbase. I suggested that you pull gora trunk, manually change the
> hbase dependency, deal with any compile/test problems, package the
> gora-hbase module into a jar then copy this over to your Nutch 2.X
> installation...
>
> Make sense?
> hth
> Lewis
--
Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Ryan,
On Mon, Aug 13, 2012 at 7:36 PM, Ryan L. Sun <li...@gmail.com> wrote:
> Nutch 2.0 is pre-configed with gora 0.2, and I checked gora website,
> 0.2.1 is the latest release.
> I edited ivy/ivy.xml to update gora-core/ gora-hbase version from 0.2
> to 0.2.1 and then re-compile. It still fetched hbase-0.90.4.jar
>
yes the HBase jar is a transitive dependency which is pulled through
gora-hbase. I suggested that you pull gora trunk, manually change the
hbase dependency, deal with any compile/test problems, package the
gora-hbase module into a jar then copy this over to your Nutch 2.X
installation...
Make sense?
hth
Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by al...@aim.com.
I tried to upgrade to hbase-0.94.0 from hbase-0.92.1 , Started hbase-0.94.0 and forget to replace hbase-0.92.1.jar with the new one:). With this config inject worked fine. But when I replaced old jar( hbase-0.92.1.jar) with new one. hbase-0.94..0.jar I get the same error as you.
Hope this will help.
Alex.
-----Original Message-----
From: Ryan L. Sun <li...@gmail.com>
To: user <us...@nutch.apache.org>
Sent: Mon, Aug 13, 2012 11:43 am
Subject: Re: nutch 2.0 with hbase 0.94.0
Nutch 2.0 is pre-configed with gora 0.2, and I checked gora website,
0.2.1 is the latest release.
I edited ivy/ivy.xml to update gora-core/ gora-hbase version from 0.2
to 0.2.1 and then re-compile. It still fetched hbase-0.90.4.jar
To Alex,
Yes, i did removed the old hbase jar from runtime/local/lib. When i
re-run it, got the following exception:
InjectorJob: starting
InjectorJob: urlDir: seed
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
at org.apache.gora.hbase.store.HBaseMapping$HBaseMappingBuilder.addFamilyProps(HBaseMapping.java:114)
at org.apache.gora.hbase.store.HBaseStore.readMapping(HBaseStore.java:543)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:111)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:69)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:288)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:298)
On Mon, Aug 13, 2012 at 10:16 AM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> Nutch contains no knowledge of which specific version of a backend you
> are using. This is however done through the gora-* dependencies
> managed by Ivy.
>
> Although this is a pretty convoluted way to do things, the best way to
> find this would be to check out Gora trunk [0], upgrade the hbase
> dependencies to whatever you need, compile and package the project
> then copy the relevant jar's over to your Nutch installation. This way
> you could run a standalone (development) hbase server and try running
> your Nutch configuration that way...
>
> hth
>
> Lewis
>
> [0] http://svn.apache.org/repos/asf/gora/trunk/
>
> On Mon, Aug 13, 2012 at 6:11 PM, Ryan L. Sun <li...@gmail.com> wrote:
>> hi all,
>>
>> I'm trying to set up nutch 2.0 with a existing hbase cluster (using
>> hbase 0.94.0). Since nutch 2.0 supports an older version (0.90.4) of
>> hbase, starting a nutch inject job crashed hbase daemon. Copying hbase
>> 0.94.0's lib to nutch/runtime/local/lib folder as google search hinted
>> doesn't work for me.
>> Any suggestions are appreciated. Thanks.
>>
>> PS. I couldn't downgrade the existing hbase cluster software version,
>> which is out of my hand.
>
>
>
> --
> Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by "Ryan L. Sun" <li...@gmail.com>.
Nutch 2.0 is pre-configed with gora 0.2, and I checked gora website,
0.2.1 is the latest release.
I edited ivy/ivy.xml to update gora-core/ gora-hbase version from 0.2
to 0.2.1 and then re-compile. It still fetched hbase-0.90.4.jar
To Alex,
Yes, i did removed the old hbase jar from runtime/local/lib. When i
re-run it, got the following exception:
InjectorJob: starting
InjectorJob: urlDir: seed
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
at org.apache.gora.hbase.store.HBaseMapping$HBaseMappingBuilder.addFamilyProps(HBaseMapping.java:114)
at org.apache.gora.hbase.store.HBaseStore.readMapping(HBaseStore.java:543)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:111)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:69)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:288)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:298)
On Mon, Aug 13, 2012 at 10:16 AM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> Nutch contains no knowledge of which specific version of a backend you
> are using. This is however done through the gora-* dependencies
> managed by Ivy.
>
> Although this is a pretty convoluted way to do things, the best way to
> find this would be to check out Gora trunk [0], upgrade the hbase
> dependencies to whatever you need, compile and package the project
> then copy the relevant jar's over to your Nutch installation. This way
> you could run a standalone (development) hbase server and try running
> your Nutch configuration that way...
>
> hth
>
> Lewis
>
> [0] http://svn.apache.org/repos/asf/gora/trunk/
>
> On Mon, Aug 13, 2012 at 6:11 PM, Ryan L. Sun <li...@gmail.com> wrote:
>> hi all,
>>
>> I'm trying to set up nutch 2.0 with a existing hbase cluster (using
>> hbase 0.94.0). Since nutch 2.0 supports an older version (0.90.4) of
>> hbase, starting a nutch inject job crashed hbase daemon. Copying hbase
>> 0.94.0's lib to nutch/runtime/local/lib folder as google search hinted
>> doesn't work for me.
>> Any suggestions are appreciated. Thanks.
>>
>> PS. I couldn't downgrade the existing hbase cluster software version,
>> which is out of my hand.
>
>
>
> --
> Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Nutch contains no knowledge of which specific version of a backend you
are using. This is however done through the gora-* dependencies
managed by Ivy.
Although this is a pretty convoluted way to do things, the best way to
find this would be to check out Gora trunk [0], upgrade the hbase
dependencies to whatever you need, compile and package the project
then copy the relevant jar's over to your Nutch installation. This way
you could run a standalone (development) hbase server and try running
your Nutch configuration that way...
hth
Lewis
[0] http://svn.apache.org/repos/asf/gora/trunk/
On Mon, Aug 13, 2012 at 6:11 PM, Ryan L. Sun <li...@gmail.com> wrote:
> hi all,
>
> I'm trying to set up nutch 2.0 with a existing hbase cluster (using
> hbase 0.94.0). Since nutch 2.0 supports an older version (0.90.4) of
> hbase, starting a nutch inject job crashed hbase daemon. Copying hbase
> 0.94.0's lib to nutch/runtime/local/lib folder as google search hinted
> doesn't work for me.
> Any suggestions are appreciated. Thanks.
>
> PS. I couldn't downgrade the existing hbase cluster software version,
> which is out of my hand.
--
Lewis
Re: nutch 2.0 with hbase 0.94.0
Posted by James <90...@qq.com>.
you can use hbase0.90.5
--
View this message in context: http://lucene.472066.n3.nabble.com/nutch-2-0-with-hbase-0-94-0-tp4000924p4015482.html
Sent from the Nutch - User mailing list archive at Nabble.com.