You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Earl Cahill <ca...@yahoo.com> on 2005/08/31 08:47:04 UTC
null lang bug? and patch?
I am in trunk, trying to do the following
bin/nutch crawl urls -dir crawl.test
where urls contains
http://spack.net/
and conf/crawl-urlfilter.txt contains
+^http://spack.net/
when I run the command I get
Exception in thread "main"
java.lang.NullPointerException
at
org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:148)
at
org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:262)
at
org.apache.nutch.tools.CrawlTool.main(CrawlTool.java:153)
I did a little digging and it appears that lang ends
up being null (couldn't quite track down where lang
should have been set). Not sure if it is a proper
fix, but changing doc.getField("lang").stringValue()
to doc.get("lang"), makes my little crawl complete.
As did commenting out that LOG.info command. Like I
say, not sure why lang is null, but if it is going to
be null, probably shouldn't be calling stringValue()
on it. Guess it didn't like http://spack.net/
Patch follows.
Earl
Index: Index:
trunk/src/java/org/apache/nutch/indexer/IndexSegment.java
===================================================================
---
trunk/src/java/org/apache/nutch/indexer/IndexSegment.java
(revision 264952)
+++
trunk/src/java/org/apache/nutch/indexer/IndexSegment.java
(working copy)
@@ -146,7 +146,7 @@
// add the document to the index
NutchAnalyzer analyzer =
AnalyzerFactory.get(doc.get("lang"));
LOG.info(" Indexing [" +
doc.getField("url").stringValue() +
- "] with analyzer " + analyzer
+ " (" + doc.getField("lang").stringValue() + ")");
+ "] with analyzer " + analyzer
+ " (" + doc.get("lang") + ")");
//LOG.info(" Doc is " + doc);
writer.addDocument(doc, analyzer);
if (count > 0 && count % LOG_STEP == 0) {
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: null lang bug? and patch?
Posted by Piotr Kosiorowski <pk...@gmail.com>.
Great - I just thought that it would be better if you look at it -
instead of me digging into the code. I wanted to be on the safe side
with 0.7.1 release.
Regards
Piotr
Jérôme Charron wrote:
>>I am a bit lost but just a quick check - shouldn't it also be committed
>>in Release-0.7 branch?
>
>
> No, the analyzer extension-point is commited only in trunk.
> It's a new feature, so I follow Committer's Rules (
> http://wiki.apache.org/nutch/Committer's_Rules)
> ;-)
> Regards
>
> Jérôme
>
Re: null lang bug? and patch?
Posted by Jérôme Charron <je...@gmail.com>.
> I am a bit lost but just a quick check - shouldn't it also be committed
> in Release-0.7 branch?
No, the analyzer extension-point is commited only in trunk.
It's a new feature, so I follow Committer's Rules (
http://wiki.apache.org/nutch/Committer's_Rules)
;-)
Regards
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
Re: null lang bug? and patch?
Posted by Piotr Kosiorowski <pk...@gmail.com>.
Hi Jérôme,
I am a bit lost but just a quick check - shouldn't it also be committed
in Release-0.7 branch?
Regards
P.
> lang is null cause you don't have activated the languageidentifier plugin.
> While testing, I always activate it, so I simply miss this potential NPE.
> Thanks for reporting problem / solution.
> The correction is committed in trunk (
> http://svn.apache.org/viewcvs.cgi?rev=264964&view=rev)
>
> Regards
>
> Jérôme
>
Re: null lang bug? and patch?
Posted by Jérôme Charron <je...@gmail.com>.
> I did a little digging and it appears that lang ends
> up being null (couldn't quite track down where lang
> should have been set). Not sure if it is a proper
> fix, but changing doc.getField("lang").stringValue()
> to doc.get("lang"), makes my little crawl complete.
lang is null cause you don't have activated the languageidentifier plugin.
While testing, I always activate it, so I simply miss this potential NPE.
Thanks for reporting problem / solution.
The correction is committed in trunk (
http://svn.apache.org/viewcvs.cgi?rev=264964&view=rev)
Regards
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/