You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Nicolás Lichtmaier <ni...@reloco.com.ar> on 2006/11/16 19:34:53 UTC
Written a plugin: now nutch fails with an error
I've written a plugin and now nutch fails with an error:
# bin/nutch generate /var/lib/tomcat/crawl/crawldb
/var/lib/tomcat/crawl/segments -topN 100
topN: 100
Generator: starting
Generator: segment: /var/lib/tomcat/crawl/segments/20061116101759
Generator: Selecting best-scoring urls due for fetch.
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:357)
at org.apache.nutch.crawl.Generator.generate(Generator.java:319)
at org.apache.nutch.crawl.Generator.main(Generator.java:395)
But I don't know how to debug this? How do I see more debug output?
Any help would be appreciated. Thanks!
Re: Written a plugin: now nutch fails with an error
Posted by Alvaro Cabrerizo <to...@gmail.com>.
2006/11/21, Nicolás Lichtmaier <ni...@reloco.com.ar>:
>
>
> >>> nutch/hadoop are you using? conf/log4j.properties file determines
> >>> the logging factor. Maybe setting logging level to INFO or DEBUG may
> >>> help.
> >> Oh, sorry =). I was trying to index some extra field I calculate from
> >> the URLs. So I followed the instructions at
> >> http://wiki.apache.org/nutch/CreateNewFilter .
> >>
> >> I've created two classes: one implementing IndexingFilter and another
> >> extending RawFieldQueryFilter. And this is the plugin.xml file:
> > Did you added my-plugin to the src/plugin/build.xml file. If not add
> > it to there, and recompile. And check the class files in the build dir
> > : check for com.pierpoint/.. if they exist.
>
> I just compiled it, made a jar and put it in an existing nutch
> installation. Then I've added it to the regex in the "plugin.includes"
> property in nutch-site.xml
>
> > However, it seems that your error might have been caused from another
> > reason. Since the generator runs neither indexingFilters nor
> > QueryFilters. If you added another plugin (like url normalizer), i
> > suggest you check that one.
>
> Yes, but the real probem is not being able to see anything. Some part of
> nutch is doing something ugly: Swallowing the real error and displaying
> a generic one. That's always a bad thing to do. I've tried to enable
> some more debug info bu I've failed, I've edited Java's
> logging.properties and nutch's log4j.properties (replacing INFO for
> DEBUG), but I couldn't get nutch to be more verbose... =/
>
>
>
>
Have you try to add your own LOG lines whithin the code you want to inspect,
at least in your new plugin, re-compile and execute.
Hope it helps
Re: Written a plugin: now nutch fails with an error
Posted by Nicolás Lichtmaier <ni...@reloco.com.ar>.
>>> nutch/hadoop are you using? conf/log4j.properties file determines
>>> the logging factor. Maybe setting logging level to INFO or DEBUG may
>>> help.
>> Oh, sorry =). I was trying to index some extra field I calculate from
>> the URLs. So I followed the instructions at
>> http://wiki.apache.org/nutch/CreateNewFilter .
>>
>> I've created two classes: one implementing IndexingFilter and another
>> extending RawFieldQueryFilter. And this is the plugin.xml file:
> Did you added my-plugin to the src/plugin/build.xml file. If not add
> it to there, and recompile. And check the class files in the build dir
> : check for com.pierpoint/.. if they exist.
I just compiled it, made a jar and put it in an existing nutch
installation. Then I've added it to the regex in the "plugin.includes"
property in nutch-site.xml
> However, it seems that your error might have been caused from another
> reason. Since the generator runs neither indexingFilters nor
> QueryFilters. If you added another plugin (like url normalizer), i
> suggest you check that one.
Yes, but the real probem is not being able to see anything. Some part of
nutch is doing something ugly: Swallowing the real error and displaying
a generic one. That's always a bad thing to do. I've tried to enable
some more debug info bu I've failed, I've edited Java's
logging.properties and nutch's log4j.properties (replacing INFO for
DEBUG), but I couldn't get nutch to be more verbose... =/
Re: Written a plugin: now nutch fails with an error
Posted by Enis Soztutar <en...@gmail.com>.
Nicolás Lichtmaier wrote:
>
>> Nicolas, you could find help if you were more specific. Which plugin
>> point did you extend and which version of nutch/hadoop are you
>> using? conf/log4j.properties file determines the logging factor.
>> Maybe setting logging level to INFO or DEBUG may help.
>
> Oh, sorry =). I was trying to index some extra field I calculate from
> the URLs. So I followed the instructions at
> http://wiki.apache.org/nutch/CreateNewFilter .
>
> I've created two classes: one implementing IndexingFilter and another
> extending RawFieldQueryFilter. And this is the plugin.xml file:
>
Did you added my-plugin to the src/plugin/build.xml file. If not add it
to there, and recompile. And check the class files in the build dir :
check for com.pierpoint/.. if they exist.
However, it seems that your error might have been caused from another
reason. Since the generator runs neither indexingFilters nor
QueryFilters. If you added another plugin (like url normalizer), i
suggest you check that one.
Re: Written a plugin: now nutch fails with an error
Posted by Nicolás Lichtmaier <ni...@reloco.com.ar>.
> Nicolas, you could find help if you were more specific. Which plugin
> point did you extend and which version of nutch/hadoop are you using?
> conf/log4j.properties file determines the logging factor. Maybe
> setting logging level to INFO or DEBUG may help.
Oh, sorry =). I was trying to index some extra field I calculate from
the URLs. So I followed the instructions at
http://wiki.apache.org/nutch/CreateNewFilter .
I've created two classes: one implementing IndexingFilter and another
extending RawFieldQueryFilter. And this is the plugin.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<plugin
id="my-plugin"
name="My Plug-in"
version="1.0.0"
provider-name="com.pierpoint">
<runtime>
<library name="my-plugin.jar">
<export name="*"/>
</library>
</runtime>
<requires>
<import plugin="nutch-extensionpoints"/>
</requires>
<extension id="com.pierpoint.datahunter.nutch.analyzer"
name="DataHunter Filter"
point="org.apache.nutch.indexer.IndexingFilter">
<implementation
id="com.pierpoint.datahunter.nutch.DataHunterIndexingFilter"
class="com.pierpoint.datahunter.nutch.DataHunterIndexingFilter">
</implementation>
</extension>
<extension id="com.pierpoint.datahunter.nutch.filter"
name="Zip3 Query Filter"
point="org.apache.nutch.searcher.site.SiteQueryFilter">
<implementation id="com.pierpoint.datahunter.nutch.Zip3QueryFilter"
class="com.pierpoint.datahunter.nutch.Zip3QueryFilter">
</implementation>
</extension>
</plugin>
Nutch version is the latest: 0.8.1.
Thanks!
Re: Written a plugin: now nutch fails with an error
Posted by Enis Soztutar <en...@gmail.com>.
Nicolás Lichtmaier wrote:
> I've written a plugin and now nutch fails with an error:
>
> # bin/nutch generate /var/lib/tomcat/crawl/crawldb
> /var/lib/tomcat/crawl/segments -topN 100
> topN: 100
> Generator: starting
> Generator: segment: /var/lib/tomcat/crawl/segments/20061116101759
> Generator: Selecting best-scoring urls due for fetch.
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:357)
> at org.apache.nutch.crawl.Generator.generate(Generator.java:319)
> at org.apache.nutch.crawl.Generator.main(Generator.java:395)
>
> But I don't know how to debug this? How do I see more debug output?
>
> Any help would be appreciated. Thanks!
>
>
Nicolas, you could find help if you were more specific. Which plugin
point did you extend and which version of nutch/hadoop are you using?
conf/log4j.properties file determines the logging factor. Maybe setting
logging level to INFO or DEBUG may help.