You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Julien Nioche <li...@gmail.com> on 2010/09/08 12:53:41 UTC
Re: Nutch 2.0 Help
Hi guys,
I've summarized the steps to follow for having GORA+Hbase with Nutch 2.0 on
http://wiki.apache.org/nutch/GORA_HBase
Feel free to amend and improve as you see fit.
Please bear in mind that Nutch 2.0 is at a very early stage and is far from
being bug-proof, see in particular [1].
HTH
Julien
[1] https://issues.apache.org/jira/browse/NUTCH-893
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
On 6 September 2010 13:35, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 2010-09-05 14:56, David Stuart wrote:
>
>> Hi All,
>>
>> I have done as per below and can create a table from within the hbase
>> shell. I found the appropriate create table method
>> bin/nutch org.apache.nutch.storage.WebTableCreator webtable but it only
>> returns null
>>
>> Any help would be great
>>
>
> You don't have to create a table manually - this should happen
> automatically when you first run any Nutch tool. Just make sure you have
> hbase-site.xml on your classpath in Nutch - best if you put it in your conf/
> and rebuild, so that it's packed into a job jar.
>
> Here's for example my config files that work with HBase (I don't use any
> non-standard settings for HBase, so my hbase-site.xml has no properties, but
> still it needs to be included in Nutch job jar):
>
> gora-hbase-mapping.xml:
> -------------------------------------------------------------------------
>
> <gora-orm>
>
> <table name="webtable">
> <family name="p"/> <!-- This can also have params like compression, bloom
> filters -->
> <family name="f"/>
> <family name="s"/>
> <family name="il"/>
> <family name="ol"/>
> <family name="h"/>
> <family name="mtdt"/>
> <family name="mk"/>
> </table>
>
> <class table="webtable" keyClass="java.lang.String"
> name="org.apache.nutch.storage.WebPage">
> <!-- fetch fields -->
> <field name="baseUrl" family="f" qualifier="bas"/>
> <field name="status" family="f" qualifier="st"/>
> <field name="prevFetchTime" family="f" qualifier="pts"/>
> <field name="fetchTime" family="f" qualifier="ts"/>
> <field name="fetchInterval" family="f" qualifier="fi"/>
> <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
> <field name="reprUrl" family="f" qualifier="rpr"/>
> <field name="content" family="f" qualifier="cnt"/>
> <field name="contentType" family="f" qualifier="typ"/>
> <field name="protocolStatus" family="f" qualifier="prot"/>
> <field name="modifiedTime" family="f" qualifier="mod"/>
>
> <!-- parse fields -->
> <field name="title" family="p" qualifier="t"/>
> <field name="text" family="p" qualifier="c"/>
> <field name="parseStatus" family="p" qualifier="st"/>
> <field name="signature" family="p" qualifier="sig"/>
> <field name="prevSignature" family="p" qualifier="psig"/>
>
> <!-- score fields -->
> <field name="score" family="s" qualifier="s"/>
>
> <field name="headers" family="h"/>
>
> <field name="inlinks" family="il"/>
>
> <field name="outlinks" family="ol"/>
>
> <field name="metadata" family="mtdt"/>
>
> <field name="markers" family="mk"/>
>
> </class>
>
> </gora-orm>
> -------------------------------------------------------------------------
>
> nutch-site.xml:
> -------------------------------------------------------------------------
> ... blah blah, a lot of unrelated stuff...
>
> <property>
> <name>storage.data.store.class</name>
> <value>org.gora.hbase.store.HBaseStore</value>
>
> <description>Default class for storing data</description>
> </property>
> -------------------------------------------------------------------------
>
> Of course you need also to use the same hadoop files (hdfs-site and
> mapred-site) as the ones that HBase uses.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
Re: Nutch 2.0 Help
Posted by Enis Soztutar <en...@gmail.com>.
Hi,
I think we need to commit all the necessary files to nutch so that it can
work out of the box for sql, hbase and casssandra. We can even write
commented-out entries in gora.properties, nutch-site.xml, etc so that using
nutch with different backends becomes a configuration change. I will open a
issue to track this down.
Cheers,
Enis
On Wed, Sep 8, 2010 at 1:53 PM, Julien Nioche <lists.digitalpebble@gmail.com
> wrote:
> Hi guys,
>
> I've summarized the steps to follow for having GORA+Hbase with Nutch 2.0 on
> http://wiki.apache.org/nutch/GORA_HBase
>
> Feel free to amend and improve as you see fit.
>
> Please bear in mind that Nutch 2.0 is at a very early stage and is far from
> being bug-proof, see in particular [1].
>
> HTH
>
> Julien
>
> [1] https://issues.apache.org/jira/browse/NUTCH-893
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
>
>
> On 6 September 2010 13:35, Andrzej Bialecki <ab...@getopt.org> wrote:
>
> > On 2010-09-05 14:56, David Stuart wrote:
> >
> >> Hi All,
> >>
> >> I have done as per below and can create a table from within the hbase
> >> shell. I found the appropriate create table method
> >> bin/nutch org.apache.nutch.storage.WebTableCreator webtable but it only
> >> returns null
> >>
> >> Any help would be great
> >>
> >
> > You don't have to create a table manually - this should happen
> > automatically when you first run any Nutch tool. Just make sure you have
> > hbase-site.xml on your classpath in Nutch - best if you put it in your
> conf/
> > and rebuild, so that it's packed into a job jar.
> >
> > Here's for example my config files that work with HBase (I don't use any
> > non-standard settings for HBase, so my hbase-site.xml has no properties,
> but
> > still it needs to be included in Nutch job jar):
> >
> > gora-hbase-mapping.xml:
> > -------------------------------------------------------------------------
> >
> > <gora-orm>
> >
> > <table name="webtable">
> > <family name="p"/> <!-- This can also have params like compression,
> bloom
> > filters -->
> > <family name="f"/>
> > <family name="s"/>
> > <family name="il"/>
> > <family name="ol"/>
> > <family name="h"/>
> > <family name="mtdt"/>
> > <family name="mk"/>
> > </table>
> >
> > <class table="webtable" keyClass="java.lang.String"
> > name="org.apache.nutch.storage.WebPage">
> > <!-- fetch fields -->
> > <field name="baseUrl" family="f" qualifier="bas"/>
> > <field name="status" family="f" qualifier="st"/>
> > <field name="prevFetchTime" family="f" qualifier="pts"/>
> > <field name="fetchTime" family="f" qualifier="ts"/>
> > <field name="fetchInterval" family="f" qualifier="fi"/>
> > <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
> > <field name="reprUrl" family="f" qualifier="rpr"/>
> > <field name="content" family="f" qualifier="cnt"/>
> > <field name="contentType" family="f" qualifier="typ"/>
> > <field name="protocolStatus" family="f" qualifier="prot"/>
> > <field name="modifiedTime" family="f" qualifier="mod"/>
> >
> > <!-- parse fields -->
> > <field name="title" family="p" qualifier="t"/>
> > <field name="text" family="p" qualifier="c"/>
> > <field name="parseStatus" family="p" qualifier="st"/>
> > <field name="signature" family="p" qualifier="sig"/>
> > <field name="prevSignature" family="p" qualifier="psig"/>
> >
> > <!-- score fields -->
> > <field name="score" family="s" qualifier="s"/>
> >
> > <field name="headers" family="h"/>
> >
> > <field name="inlinks" family="il"/>
> >
> > <field name="outlinks" family="ol"/>
> >
> > <field name="metadata" family="mtdt"/>
> >
> > <field name="markers" family="mk"/>
> >
> > </class>
> >
> > </gora-orm>
> > -------------------------------------------------------------------------
> >
> > nutch-site.xml:
> > -------------------------------------------------------------------------
> > ... blah blah, a lot of unrelated stuff...
> >
> > <property>
> > <name>storage.data.store.class</name>
> > <value>org.gora.hbase.store.HBaseStore</value>
> >
> > <description>Default class for storing data</description>
> > </property>
> > -------------------------------------------------------------------------
> >
> > Of course you need also to use the same hadoop files (hdfs-site and
> > mapred-site) as the ones that HBase uses.
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >
>
Re: Nutch 2.0 Help
Posted by Enis Soztutar <en...@gmail.com>.
Hi,
I think we need to commit all the necessary files to nutch so that it can
work out of the box for sql, hbase and casssandra. We can even write
commented-out entries in gora.properties, nutch-site.xml, etc so that using
nutch with different backends becomes a configuration change. I will open a
issue to track this down.
Cheers,
Enis
On Wed, Sep 8, 2010 at 1:53 PM, Julien Nioche <lists.digitalpebble@gmail.com
> wrote:
> Hi guys,
>
> I've summarized the steps to follow for having GORA+Hbase with Nutch 2.0 on
> http://wiki.apache.org/nutch/GORA_HBase
>
> Feel free to amend and improve as you see fit.
>
> Please bear in mind that Nutch 2.0 is at a very early stage and is far from
> being bug-proof, see in particular [1].
>
> HTH
>
> Julien
>
> [1] https://issues.apache.org/jira/browse/NUTCH-893
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
>
>
> On 6 September 2010 13:35, Andrzej Bialecki <ab...@getopt.org> wrote:
>
> > On 2010-09-05 14:56, David Stuart wrote:
> >
> >> Hi All,
> >>
> >> I have done as per below and can create a table from within the hbase
> >> shell. I found the appropriate create table method
> >> bin/nutch org.apache.nutch.storage.WebTableCreator webtable but it only
> >> returns null
> >>
> >> Any help would be great
> >>
> >
> > You don't have to create a table manually - this should happen
> > automatically when you first run any Nutch tool. Just make sure you have
> > hbase-site.xml on your classpath in Nutch - best if you put it in your
> conf/
> > and rebuild, so that it's packed into a job jar.
> >
> > Here's for example my config files that work with HBase (I don't use any
> > non-standard settings for HBase, so my hbase-site.xml has no properties,
> but
> > still it needs to be included in Nutch job jar):
> >
> > gora-hbase-mapping.xml:
> > -------------------------------------------------------------------------
> >
> > <gora-orm>
> >
> > <table name="webtable">
> > <family name="p"/> <!-- This can also have params like compression,
> bloom
> > filters -->
> > <family name="f"/>
> > <family name="s"/>
> > <family name="il"/>
> > <family name="ol"/>
> > <family name="h"/>
> > <family name="mtdt"/>
> > <family name="mk"/>
> > </table>
> >
> > <class table="webtable" keyClass="java.lang.String"
> > name="org.apache.nutch.storage.WebPage">
> > <!-- fetch fields -->
> > <field name="baseUrl" family="f" qualifier="bas"/>
> > <field name="status" family="f" qualifier="st"/>
> > <field name="prevFetchTime" family="f" qualifier="pts"/>
> > <field name="fetchTime" family="f" qualifier="ts"/>
> > <field name="fetchInterval" family="f" qualifier="fi"/>
> > <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
> > <field name="reprUrl" family="f" qualifier="rpr"/>
> > <field name="content" family="f" qualifier="cnt"/>
> > <field name="contentType" family="f" qualifier="typ"/>
> > <field name="protocolStatus" family="f" qualifier="prot"/>
> > <field name="modifiedTime" family="f" qualifier="mod"/>
> >
> > <!-- parse fields -->
> > <field name="title" family="p" qualifier="t"/>
> > <field name="text" family="p" qualifier="c"/>
> > <field name="parseStatus" family="p" qualifier="st"/>
> > <field name="signature" family="p" qualifier="sig"/>
> > <field name="prevSignature" family="p" qualifier="psig"/>
> >
> > <!-- score fields -->
> > <field name="score" family="s" qualifier="s"/>
> >
> > <field name="headers" family="h"/>
> >
> > <field name="inlinks" family="il"/>
> >
> > <field name="outlinks" family="ol"/>
> >
> > <field name="metadata" family="mtdt"/>
> >
> > <field name="markers" family="mk"/>
> >
> > </class>
> >
> > </gora-orm>
> > -------------------------------------------------------------------------
> >
> > nutch-site.xml:
> > -------------------------------------------------------------------------
> > ... blah blah, a lot of unrelated stuff...
> >
> > <property>
> > <name>storage.data.store.class</name>
> > <value>org.gora.hbase.store.HBaseStore</value>
> >
> > <description>Default class for storing data</description>
> > </property>
> > -------------------------------------------------------------------------
> >
> > Of course you need also to use the same hadoop files (hdfs-site and
> > mapred-site) as the ones that HBase uses.
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >
>