Nailing Down Nutch Parser Plugin Configuration

[jc...@pureperfect.com]

Hi,


I'm a nutch newbie and I want to be able to access the content of
documents as nutch is crawling them, do a little parsing and write the
values to a database. I chose to implement the Parser extension point as
it seemed the most logical place to do the parsing (not sure if this is
the best solution). 


The issue I have is that there is likely something wrong with my
configuration as it doesn't seem like the plugin is getting executed.
When I look at logs/hadoop.log for "my-plugin", it looks like it is
getting loaded in the plugin repository, but not attached to an
extension point:


~~~

2012-06-18 20:57:19,912 INFO  plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository - Registered
Plugins:
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         the
nutch core extension points (nutch-extensionpoints)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Basic
URL Normalizer (urlnormalizer-basic)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Html
Parse Plug-in (parse-html)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Basic
Indexing Filter (index-basic)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         HTTP
Framework (lib-http)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -        
Pass-through URL Normalizer (urlnormalizer-pass)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Regex
URL Filter (urlfilter-regex)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Http
Protocol Plug-in (protocol-http)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Regex
URL Normalizer (urlnormalizer-regex)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Tika
Parser Plug-in (parse-tika)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         OPIC
Scoring Plug-in (scoring-opic)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -        
CyberNeko HTML Parser (lib-nekohtml)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Anchor
Indexing Filter (index-anchor)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         MyPlugin
(my-plugin)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Regex
URL Filter Framework (lib-regex-filter)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository - Registered
Extension-Points:
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2012-06-18 20:57:19,912 INFO  plugin.PluginRepository -         Nutch
Content Parser (org.apache.nutch.parse.Parser)
2012-06-18 20:57:19,913 INFO  plugin.PluginRepository -         Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2012-06-18 20:57:19,937 INFO  regex.RegexURLNormalizer - can't find
rules for scope 'inject', using default
2012-06-18 20:57:26,730 INFO  crawl.Injector - Injector: Merging
injected urls into crawl db.
~~~


here is my plugin.xml file:


~~~

<plugin
        id="my-plugin"
        name="MyPlugin"
        version="0.1"
        provider-name="nutch.org">

    <runtime>
        <library name="my-plugin-0.1-SNAPSHOT.jar">
            <export name="*"/>
        </library>
    </runtime>

    <requires>
        <import plugin="nutch-extensionpoints"/>
        <!--
        <import plugin="lib-nekohtml"/>
        -->
    </requires>

    <extension id="org.nutch.mine"
                            name="MyPlugin"
                            point="org.apache.nutch.parse.Parser">

    <implementation id="org.nutch.mine.MyPlugin"
                    class="org.nutch.mine.MyPlugin">
        <parameter name="contentType"
value="text/html|application/xhtml+xml"/>
        <parameter name="pathSuffix" value=""/>
    </implementation>
</extension>
</plugin>
~~~~                  



Any help would be greatly appreciated.