You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2005/08/10 15:14:37 UTC
injecting outlinks?
Is it possible to inject outlinks into the ParseData stored within a
segment of an already fetched/parsed page? After a bit of digging
into the code, I'm not seeing anything to make this possible yet.
My use case is that I want to crawl a sites with RDF metadata behind
the scenes. Some of the sites I'll crawl will have the link to
metadata physically in the HTML, but I'd like to provide a back door
for sites that have metadata but have not added links to it from the
HTML yet.
For the moment, I'll create a custom HTML parser that will
automatically inject them - but I was wondering if there was a more
direct way to affect the ParseData outlinks so that the next fetch
will pick them up.
Thanks,
Erik
Re: injecting outlinks?
Posted by Piotr Kosiorowski <pk...@gmail.com>.
It would be better to wait till next week - according to ealier
discussion on nutch-dev deadline for code changes for 0.7 release was on
Aug 8.
And the release is planned on Aug 15. So I would wait with everything
not connected with release.
Regards
Piotr
Erik Hatcher wrote:
>
> On Aug 10, 2005, at 4:16 PM, Doug Cutting wrote:
>
>> Erik Hatcher wrote:
>>
>>> Nevermind.... I had removed parse-html plugin from my nutch-
>>> site.xml inadvertently, which caused this issue.
>>> However, it'd be nice to get a clearer error message for this
>>> situation. Thoughts?
>>>
>>
>> Perhaps in ParserFactory a synchronized HashMap should be used in
>> place of a Hashtable, so that a null parser for a content type can
>> still be cached. Then Fetcher.java should be changed to gracefully
>> turn a null parser into a ParseStatus.STATUS_NOTPARSED. Does that
>> sound right?
>
>
> Seems good to me, at least the Fetcher.java change. Is the
> synchronized HashMap change necessary for the Fetcher.java change to
> work properly?
>
> I can probably make these changes and commit them. But I'm wondering
> if there will be merge pains for the MapReduce branch with trunk
> changes like this.
>
> Erik
>
>
Re: injecting outlinks?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 10, 2005, at 4:16 PM, Doug Cutting wrote:
> Erik Hatcher wrote:
>
>> Nevermind.... I had removed parse-html plugin from my nutch-
>> site.xml inadvertently, which caused this issue.
>> However, it'd be nice to get a clearer error message for this
>> situation. Thoughts?
>>
>
> Perhaps in ParserFactory a synchronized HashMap should be used in
> place of a Hashtable, so that a null parser for a content type can
> still be cached. Then Fetcher.java should be changed to gracefully
> turn a null parser into a ParseStatus.STATUS_NOTPARSED. Does that
> sound right?
Seems good to me, at least the Fetcher.java change. Is the
synchronized HashMap change necessary for the Fetcher.java change to
work properly?
I can probably make these changes and commit them. But I'm
wondering if there will be merge pains for the MapReduce branch with
trunk changes like this.
Erik
Re: injecting outlinks?
Posted by Doug Cutting <cu...@nutch.org>.
Erik Hatcher wrote:
> Nevermind.... I had removed parse-html plugin from my nutch-site.xml
> inadvertently, which caused this issue.
>
> However, it'd be nice to get a clearer error message for this
> situation. Thoughts?
Perhaps in ParserFactory a synchronized HashMap should be used in place
of a Hashtable, so that a null parser for a content type can still be
cached. Then Fetcher.java should be changed to gracefully turn a null
parser into a ParseStatus.STATUS_NOTPARSED. Does that sound right?
Doug
>
> On Aug 10, 2005, at 2:25 PM, Erik Hatcher wrote:
>
>> On Aug 10, 2005, at 11:51 AM, Andrzej Bialecki wrote:
>>
>>> There is a plugin hook in HTML parser, where it calls HTML filters
>>> (HtmlParser.java:207). These filters can add/modify anything
>>> collected so far. You could implement an HTMLFilter plugin similar
>>> to the creativecommons plugin, which would be automatically called
>>> here to add outlinks.
>>>
>>
>> I had it working with a custom HTML parser, but after your mail I
>> changed to the filter approach as that is much cleaner. However, I'm
>> now getting this when running a fetch:
>>
>> 050810 142101 parsing: /Users/erik/dev/arp/nines/build/plugins/
>> nines-rdfmap/plugin.xml
>> 050810 142101 impl: point=org.apache.nutch.parse.HtmlParseFilter
>> class=org.nines.nutch.RdfLinkMapFilter
>> 050810 142102 Configured Client
>> java.lang.NullPointerException
>> at java.util.Hashtable.put(Hashtable.java:396)
>> at org.apache.nutch.parse.ParserFactory.getExtension
>> (ParserFactory.java:91)
>> at org.apache.nutch.parse.ParserFactory.getParser
>> (ParserFactory.java:59)
>> at org.apache.nutch.fetcher.Fetcher
>> $FetcherThread.handleFetch(Fetcher.java:253)
>> at org.apache.nutch.fetcher.Fetcher$FetcherThread.run
>> (Fetcher.java:148)
>> 050810 142104 fetch okay, but can't parse http://
>> www.rossettiarchive.org/docs/1-1847.s244.raw.html, reason: failed
>> (2,200): java.lang.NullPointerException
>>
>> My plugin.xml looks like this:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <plugin
>> id="nines-rdfmap"
>> name="NINES HTML Parse Filter"
>> version="1.0.0"
>> provider-name="nines.org">
>>
>> <extension-point
>> id="org.apache.nutch.parse.HtmlParseFilter"
>> name="HTML Parse Filter"/>
>>
>>
>> <runtime>
>> <library name="nines-rdfmap.jar">
>> <export name="*"/>
>> </library>
>> </runtime>
>>
>> <extension id="org.nines.nutch.RdfLinkMapFilter"
>> name="RDF Link Map Filter"
>> point="org.apache.nutch.parse.HtmlParseFilter">
>> <implementation id="RdfLinkMapFilter"
>> class="org.nines.nutch.RdfLinkMapFilter"/>
>> </extension>
>>
>> </plugin>
>>
>> I'm sure something simple is awry, but I'm not seeing it currently.
>> What am I missing?
>>
>> Perhaps some improvements to the error message could be made to help
>> out in this type of scenario?
>>
>> Thanks,
>> Erik
>>
>>
>
Re: injecting outlinks?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Nevermind.... I had removed parse-html plugin from my nutch-site.xml
inadvertently, which caused this issue.
However, it'd be nice to get a clearer error message for this
situation. Thoughts?
Erik
On Aug 10, 2005, at 2:25 PM, Erik Hatcher wrote:
> On Aug 10, 2005, at 11:51 AM, Andrzej Bialecki wrote:
>
>> There is a plugin hook in HTML parser, where it calls HTML filters
>> (HtmlParser.java:207). These filters can add/modify anything
>> collected so far. You could implement an HTMLFilter plugin similar
>> to the creativecommons plugin, which would be automatically called
>> here to add outlinks.
>>
>
> I had it working with a custom HTML parser, but after your mail I
> changed to the filter approach as that is much cleaner. However,
> I'm now getting this when running a fetch:
>
> 050810 142101 parsing: /Users/erik/dev/arp/nines/build/plugins/
> nines-rdfmap/plugin.xml
> 050810 142101 impl: point=org.apache.nutch.parse.HtmlParseFilter
> class=org.nines.nutch.RdfLinkMapFilter
> 050810 142102 Configured Client
> java.lang.NullPointerException
> at java.util.Hashtable.put(Hashtable.java:396)
> at org.apache.nutch.parse.ParserFactory.getExtension
> (ParserFactory.java:91)
> at org.apache.nutch.parse.ParserFactory.getParser
> (ParserFactory.java:59)
> at org.apache.nutch.fetcher.Fetcher
> $FetcherThread.handleFetch(Fetcher.java:253)
> at org.apache.nutch.fetcher.Fetcher$FetcherThread.run
> (Fetcher.java:148)
> 050810 142104 fetch okay, but can't parse http://
> www.rossettiarchive.org/docs/1-1847.s244.raw.html, reason: failed
> (2,200): java.lang.NullPointerException
>
> My plugin.xml looks like this:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <plugin
> id="nines-rdfmap"
> name="NINES HTML Parse Filter"
> version="1.0.0"
> provider-name="nines.org">
>
> <extension-point
> id="org.apache.nutch.parse.HtmlParseFilter"
> name="HTML Parse Filter"/>
>
>
> <runtime>
> <library name="nines-rdfmap.jar">
> <export name="*"/>
> </library>
> </runtime>
>
> <extension id="org.nines.nutch.RdfLinkMapFilter"
> name="RDF Link Map Filter"
> point="org.apache.nutch.parse.HtmlParseFilter">
> <implementation id="RdfLinkMapFilter"
> class="org.nines.nutch.RdfLinkMapFilter"/>
> </extension>
>
> </plugin>
>
> I'm sure something simple is awry, but I'm not seeing it
> currently. What am I missing?
>
> Perhaps some improvements to the error message could be made to
> help out in this type of scenario?
>
> Thanks,
> Erik
>
>
Re: injecting outlinks?
Posted by Andrzej Bialecki <ab...@getopt.org>.
Erik Hatcher wrote:
> <extension id="org.nines.nutch.RdfLinkMapFilter"
> name="RDF Link Map Filter"
> point="org.apache.nutch.parse.HtmlParseFilter">
> <implementation id="RdfLinkMapFilter"
> class="org.nines.nutch.RdfLinkMapFilter"/>
You need to add pathSuffix (may be empty) and contentType.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: injecting outlinks?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 10, 2005, at 11:51 AM, Andrzej Bialecki wrote:
> There is a plugin hook in HTML parser, where it calls HTML filters
> (HtmlParser.java:207). These filters can add/modify anything
> collected so far. You could implement an HTMLFilter plugin similar
> to the creativecommons plugin, which would be automatically called
> here to add outlinks.
I had it working with a custom HTML parser, but after your mail I
changed to the filter approach as that is much cleaner. However, I'm
now getting this when running a fetch:
050810 142101 parsing: /Users/erik/dev/arp/nines/build/plugins/nines-
rdfmap/plugin.xml
050810 142101 impl: point=org.apache.nutch.parse.HtmlParseFilter
class=org.nines.nutch.RdfLinkMapFilter
050810 142102 Configured Client
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:396)
at org.apache.nutch.parse.ParserFactory.getExtension
(ParserFactory.java:91)
at org.apache.nutch.parse.ParserFactory.getParser
(ParserFactory.java:59)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch
(Fetcher.java:253)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.run
(Fetcher.java:148)
050810 142104 fetch okay, but can't parse http://
www.rossettiarchive.org/docs/1-1847.s244.raw.html, reason: failed
(2,200): java.lang.NullPointerException
My plugin.xml looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<plugin
id="nines-rdfmap"
name="NINES HTML Parse Filter"
version="1.0.0"
provider-name="nines.org">
<extension-point
id="org.apache.nutch.parse.HtmlParseFilter"
name="HTML Parse Filter"/>
<runtime>
<library name="nines-rdfmap.jar">
<export name="*"/>
</library>
</runtime>
<extension id="org.nines.nutch.RdfLinkMapFilter"
name="RDF Link Map Filter"
point="org.apache.nutch.parse.HtmlParseFilter">
<implementation id="RdfLinkMapFilter"
class="org.nines.nutch.RdfLinkMapFilter"/>
</extension>
</plugin>
I'm sure something simple is awry, but I'm not seeing it currently.
What am I missing?
Perhaps some improvements to the error message could be made to help
out in this type of scenario?
Thanks,
Erik
RE: updatedb, index, mergesegs
Posted by EM <em...@cpuedge.com>.
Thanks!
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Wednesday, August 10, 2005 12:32 PM
To: nutch-user@lucene.apache.org
Subject: Re: updatedb, index, mergesegs
EM wrote:
> Can 'updatedb' and 'index' be performed concurrently on a segment?
>
Yes.
> Does 'mergesegs' require the segments to be index-ed or updatedb-ed before
> merging?
No and no. Mergesegs creates its own (lightweight) index to perform
deduplication. It can also create the final index at the end of merging,
if you so wish.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: updatedb, index, mergesegs
Posted by Andrzej Bialecki <ab...@getopt.org>.
EM wrote:
> Can 'updatedb' and 'index' be performed concurrently on a segment?
>
Yes.
> Does 'mergesegs' require the segments to be index-ed or updatedb-ed before
> merging?
No and no. Mergesegs creates its own (lightweight) index to perform
deduplication. It can also create the final index at the end of merging,
if you so wish.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
updatedb, index, mergesegs
Posted by EM <em...@cpuedge.com>.
Can 'updatedb' and 'index' be performed concurrently on a segment?
Does 'mergesegs' require the segments to be index-ed or updatedb-ed before
merging?
Re: injecting outlinks?
Posted by Andrzej Bialecki <ab...@getopt.org>.
Erik Hatcher wrote:
> Is it possible to inject outlinks into the ParseData stored within a
> segment of an already fetched/parsed page? After a bit of digging
> into the code, I'm not seeing anything to make this possible yet.
>
I'm assuming you mean "during parsing" - segment data written to disk is
treated as immutable, i.e. there are no tools to modify it on-disk.
> My use case is that I want to crawl a sites with RDF metadata behind
> the scenes. Some of the sites I'll crawl will have the link to
> metadata physically in the HTML, but I'd like to provide a back door
> for sites that have metadata but have not added links to it from the
> HTML yet.
>
> For the moment, I'll create a custom HTML parser that will
> automatically inject them - but I was wondering if there was a more
> direct way to affect the ParseData outlinks so that the next fetch will
> pick them up.
There is a plugin hook in HTML parser, where it calls HTML filters
(HtmlParser.java:207). These filters can add/modify anything collected
so far. You could implement an HTMLFilter plugin similar to the
creativecommons plugin, which would be automatically called here to add
outlinks.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com