Prevent website parts of being indexed problem

[sebio <se...@gmail.com>]

Hello nutch users,

i think thats a common problem and i found this thread: 
http://www.adick.at/2008/09/16/nutch-prevent-sections-of-a-website-from-being-indexed/
http://lucene.472066.n3.nabble.com/ignore-content-between-tags-crawl-only-between-tags-td609676.html 
and this blog entry 
http://www.adick.at/2008/09/16/nutch-prevent-sections-of-a-website-from-being-indexed/
http://www.adick.at/2008/09/16/nutch-prevent-sections-of-a-website-from-being-indexed/ 

I modified the DOMContentUtils class as suggested. When i test the plugin
using "bin/nutch plugin parse-htmlnoindex
com.example.nutch.parse.html.HtmlParser ~/test.html" everything works fine.
The result text only states the parts that are not in "nutch_noindex".

Nevertheless, when indexing a whole site the section inside "nutch_noindex"
still appears in the solr index. Assuming the modified "parse-html" plugin
is named "parse-htmlnoindex" my "nutch-sites.xml" lookes like this.

 <property>
    <name>plugin.includes</name>
   
<value>protocol-httpclient|urlfilter-regex|parse-(htmlnoindex|tika)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
  </property>

I started the test crawling/indexing process with: "bin/nutch crawl urls
-solr http://localhost:8983/solr/de -depth 2 -topN 5"

My questions are:

1. Is that the right way to index/crawl a site or do i have to include the
used plugins explicitely in the command?

2. I am wondering if this feature isn't a common use case and if such
functionalities are already available in nutch? Perhaps in nutch 2.0?

I am using nutch 1.5.1 with solr 3.4.0. Solr uses the example schema.xml
shipped with nutch. I am new to the whole nutch topic and i hope someone
could help me with that.


Kinde regards,
Sebastian



--
View this message in context: http://lucene.472066.n3.nabble.com/Prevent-website-parts-of-being-indexed-problem-tp3997213.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Prevent website parts of being indexed

[sebio <se...@gmail.com>]

Ok,

i found out that i had to change parse-default.xml and point the default
parsers to my own modified parser.

Using "bin/nutch parsechecker -dumpText
http://www.mytestsite.com/index.html" yields the correct parsed text without
the "nutch_noindex" sections.

The solr index still contains the full parsed text.




--
View this message in context: http://lucene.472066.n3.nabble.com/Prevent-website-parts-of-being-indexed-tp3997213p3997236.html
Sent from the Nutch - User mailing list archive at Nabble.com.