You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Fuad Efendi <fu...@efendi.ca> on 2005/08/10 19:39:07 UTC
How to extend Nutch
I need specific pre-processing of a html-page, to add more fields to
Document before storing it in Index, and to modify web-interface
accordingly.
Where is the base point of extension?
Thanks!
Re: [Nutch-general] How to extend Nutch
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Look at the creativecommons plugin built into Nutch. That will show
you how to do it by implementing an HtmlParseFilter. I created a
custom plugin just like this just a few moments ago and it works great.
Erik
On Aug 10, 2005, at 1:39 PM, Fuad Efendi wrote:
> I need specific pre-processing of a html-page, to add more fields to
> Document before storing it in Index, and to modify web-interface
> accordingly.
>
> Where is the base point of extension?
> Thanks!
>
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams *
> Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/
> bsce5sf
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
>
RE: [Nutch-general] How to extend Nutch
Posted by Fuad Efendi <fu...@efendi.ca>.
Ok, I see it...
It uses nutch-site.xml file from JAR file, not from CONF directory
(where I updated nutch-site.xml). Rebuild...
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: Thursday, August 11, 2005 5:12 PM
To: nutch-user@lucene.apache.org
Subject: RE: [Nutch-general] How to extend Nutch
I copided to nutch-site.xml (added CreativeCommons plugin):
<property>
<name>plugin.includes</name>
<value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-ba
sic|query-(basic|site|url)|creativecommons</value>
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins.
</description>
</property>
That's enough? Doesn't work, need help...
Thanks
-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
Sent: Wednesday, August 10, 2005 4:00 PM
To: nutch-user@lucene.apache.org
Subject: Re: [Nutch-general] How to extend Nutch
nutch-site.xml is the only config file you should touch, by copying
the appropriate section from nutch-default.xml and customizing it.
Yes, you will need to write a custom plugin like the creativecommons
one.
Erik
On Aug 10, 2005, at 2:44 PM, Fuad Efendi wrote:
>
> I probably need to work with plugins, and to modify config files... I
> need to add additional field to Document, and to show it on a web-page
>
> nutch-conf.xsl
> nutch-default.xml
> nutch-site.xml
>
> Am I right?
> Thanks
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Wednesday, August 10, 2005 2:15 PM
> To: nutch-user@lucene.apache.org
> Subject: RE: [Nutch-general] How to extend Nutch
>
>
> So, I need to modify some existing classes, isn't it?
>
>
> -----Original Message-----
> From: ogjunk-nutch@yahoo.com [mailto:ogjunk-nutch@yahoo.com]
> Sent: Wednesday, August 10, 2005 1:48 PM
> To: user@nutch.org
> Subject: Re: [Nutch-general] How to extend Nutch
>
>
> Probably IndexingFilter or HtmlParser for indexing and for indexing I
> think there is something in org.apache.nutch.search.... some class
> that
> starts with Raw.... I just saw this in the Javadoc earlier.
>
> Otis
>
> --- Fuad Efendi <fu...@efendi.ca> wrote:
>
>
>> I need specific pre-processing of a html-page, to add more fields to
>> Document before storing it in Index, and to modify web-interface
>> accordingly.
>>
>> Where is the base point of extension?
>> Thanks!
>>
>
RE: [Nutch-general] How to extend Nutch
Posted by Fuad Efendi <fu...@efendi.ca>.
I copided to nutch-site.xml (added CreativeCommons plugin):
<property>
<name>plugin.includes</name>
<value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-ba
sic|query-(basic|site|url)|creativecommons</value>
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins.
</description>
</property>
That's enough? Doesn't work, need help...
Thanks
-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
Sent: Wednesday, August 10, 2005 4:00 PM
To: nutch-user@lucene.apache.org
Subject: Re: [Nutch-general] How to extend Nutch
nutch-site.xml is the only config file you should touch, by copying
the appropriate section from nutch-default.xml and customizing it.
Yes, you will need to write a custom plugin like the creativecommons
one.
Erik
On Aug 10, 2005, at 2:44 PM, Fuad Efendi wrote:
>
> I probably need to work with plugins, and to modify config files... I
> need to add additional field to Document, and to show it on a web-page
>
> nutch-conf.xsl
> nutch-default.xml
> nutch-site.xml
>
> Am I right?
> Thanks
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Wednesday, August 10, 2005 2:15 PM
> To: nutch-user@lucene.apache.org
> Subject: RE: [Nutch-general] How to extend Nutch
>
>
> So, I need to modify some existing classes, isn't it?
>
>
> -----Original Message-----
> From: ogjunk-nutch@yahoo.com [mailto:ogjunk-nutch@yahoo.com]
> Sent: Wednesday, August 10, 2005 1:48 PM
> To: user@nutch.org
> Subject: Re: [Nutch-general] How to extend Nutch
>
>
> Probably IndexingFilter or HtmlParser for indexing and for indexing I
> think there is something in org.apache.nutch.search.... some class
> that
> starts with Raw.... I just saw this in the Javadoc earlier.
>
> Otis
>
> --- Fuad Efendi <fu...@efendi.ca> wrote:
>
>
>> I need specific pre-processing of a html-page, to add more fields to
>> Document before storing it in Index, and to modify web-interface
>> accordingly.
>>
>> Where is the base point of extension?
>> Thanks!
>>
>
Re: [Nutch-general] How to extend Nutch
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
nutch-site.xml is the only config file you should touch, by copying
the appropriate section from nutch-default.xml and customizing it.
Yes, you will need to write a custom plugin like the creativecommons
one.
Erik
On Aug 10, 2005, at 2:44 PM, Fuad Efendi wrote:
>
> I probably need to work with plugins, and to modify config files... I
> need to add additional field to Document, and to show it on a web-page
>
> nutch-conf.xsl
> nutch-default.xml
> nutch-site.xml
>
> Am I right?
> Thanks
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: Wednesday, August 10, 2005 2:15 PM
> To: nutch-user@lucene.apache.org
> Subject: RE: [Nutch-general] How to extend Nutch
>
>
> So, I need to modify some existing classes, isn't it?
>
>
> -----Original Message-----
> From: ogjunk-nutch@yahoo.com [mailto:ogjunk-nutch@yahoo.com]
> Sent: Wednesday, August 10, 2005 1:48 PM
> To: user@nutch.org
> Subject: Re: [Nutch-general] How to extend Nutch
>
>
> Probably IndexingFilter or HtmlParser for indexing and for indexing I
> think there is something in org.apache.nutch.search.... some class
> that
> starts with Raw.... I just saw this in the Javadoc earlier.
>
> Otis
>
> --- Fuad Efendi <fu...@efendi.ca> wrote:
>
>
>> I need specific pre-processing of a html-page, to add more fields to
>> Document before storing it in Index, and to modify web-interface
>> accordingly.
>>
>> Where is the base point of extension?
>> Thanks!
>>
>
RE: [Nutch-general] How to extend Nutch
Posted by Fuad Efendi <fu...@efendi.ca>.
I probably need to work with plugins, and to modify config files... I
need to add additional field to Document, and to show it on a web-page
nutch-conf.xsl
nutch-default.xml
nutch-site.xml
Am I right?
Thanks
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: Wednesday, August 10, 2005 2:15 PM
To: nutch-user@lucene.apache.org
Subject: RE: [Nutch-general] How to extend Nutch
So, I need to modify some existing classes, isn't it?
-----Original Message-----
From: ogjunk-nutch@yahoo.com [mailto:ogjunk-nutch@yahoo.com]
Sent: Wednesday, August 10, 2005 1:48 PM
To: user@nutch.org
Subject: Re: [Nutch-general] How to extend Nutch
Probably IndexingFilter or HtmlParser for indexing and for indexing I
think there is something in org.apache.nutch.search.... some class that
starts with Raw.... I just saw this in the Javadoc earlier.
Otis
--- Fuad Efendi <fu...@efendi.ca> wrote:
> I need specific pre-processing of a html-page, to add more fields to
> Document before storing it in Index, and to modify web-interface
> accordingly.
>
> Where is the base point of extension?
> Thanks!
>
RE: [Nutch-general] How to extend Nutch
Posted by Fuad Efendi <fu...@efendi.ca>.
So, I need to modify some existing classes, isn't it?
-----Original Message-----
From: ogjunk-nutch@yahoo.com [mailto:ogjunk-nutch@yahoo.com]
Sent: Wednesday, August 10, 2005 1:48 PM
To: user@nutch.org
Subject: Re: [Nutch-general] How to extend Nutch
Probably IndexingFilter or HtmlParser for indexing and for indexing I
think there is something in org.apache.nutch.search.... some class that
starts with Raw.... I just saw this in the Javadoc earlier.
Otis
--- Fuad Efendi <fu...@efendi.ca> wrote:
> I need specific pre-processing of a html-page, to add more fields to
> Document before storing it in Index, and to modify web-interface
> accordingly.
>
> Where is the base point of extension?
> Thanks!
>
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> Practices Agile & Plan-Driven Development * Managing Projects & Teams
> * Testing & QA
> Security * Process Improvement & Measurement *
> http://www.sqe.com/bsce5sf
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
>
Re: [Nutch-general] How to extend Nutch
Posted by og...@yahoo.com.
Probably IndexingFilter or HtmlParser for indexing and for indexing I
think there is something in org.apache.nutch.search.... some class that
starts with Raw.... I just saw this in the Javadoc earlier.
Otis
--- Fuad Efendi <fu...@efendi.ca> wrote:
> I need specific pre-processing of a html-page, to add more fields to
> Document before storing it in Index, and to modify web-interface
> accordingly.
>
> Where is the base point of extension?
> Thanks!
>
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing
> & QA
> Security * Process Improvement & Measurement *
> http://www.sqe.com/bsce5sf
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
>