You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2005/07/19 14:57:50 UTC
RDF plugin questions
Hi,
I'm embarking on an adventure with Nutch to crawl 19th century
digital scholarly archives (like the Rossetti Archive, where I work)
for the nines.org system. The goal is to use a normal crawl on a
selected set of sites, and extract some additional information in the
process. Many HTML pages of these archives will be tagged with a
<head> tag like this page http://www.rossettiarchive.org/docs/
1-1882.s241.raw.html:
<link type="application/rdf+xml" title="The Question (for a
Design)" href="http://www.rossettiarchive.org/docs/1-1882.s241.raw.rdf">
What I need is a facility to fetch and parse that RDF in a custom
way, such that the RDF gets dropped into an RDF engine (currently
using Kowari). At the point of processing the RDF data I want to
know the URL of the page it came from (the one containing the <link>)
such that I can add another RDF statement to the data with provenance
information.
I see that Nutch fetches the RDF (in the crawl log using the basic
crawl command). Can a parse plugin know what page the RDF link came
from? If not, then how should I craft things to get that info?
At this point I'm a newbie with Nutch, and glad to have this mailing
list for advice. I'm quite open to suggestions on how to go about
building this custom add-on to Nutch and quite willing to generalize
it and contribute it.
Thanks,
Erik
Re: [Nutch-general] Re: RDF plugin questions
Posted by Stefan Groschupf <sg...@media-style.com>.
Hi Erik,
> Stefan - thanks for the reply. I'm still digesting Nutch and how
> to work with it at a basic level but it does make sense to allow
> metadata to tag along with fetches - I certainly don't know enough
> yet to say whether your patch fits into the long-term vision of
> Nutch or not yet.
Well I would be interested to hear about the long-term vision of
Nutch as well. :)
> I've started writing a custom RDF parser plugin that will take the
> URL and simply add it to Kowari (letting Kowari actually parse it
> and ingest it). But I'm feeling like this might not be the best
> approach.
>
> At what stage would make the most sense for ingesting RDF into an
> external system? Is parsing the most logical stage?
>
> Further on this topic, I'm curious about indexing multiple
> "documents" per .rdf file fetched - for instance, one document per
> RDF "resource".
You hit another problem I see since some time. For example I see this
problem for rss parsing, image search or any other them where you
have multiple logical documents per physical document (xml feed, html
page).
> Is this currently possible with a plugin?
NO! As far I know you can only have one document per one URL.
> If not, what would it take to do something like this? Maybe this
> approach doesn't even make sense in the Nutch sense - I'm just
> exploring my architectural options.
I solved a similar problem with following steps.
I fetch but I do not parse until fetch time.
In the next step I read the unparsed content from the segment use a
own parser and directly indexed the content I had parsed. Beside this
I had written a text file with extracted URLs. This urls was merged
back to webdb in the end.
It was working but not more than a prototype and at least I was
asking myself if it makes sense to use nutch for such a task.
Anyway I would be very happy to see a patch that allows to extract
multiple documents from one source ( this would help to implement a
better rss or image search) however I think that is a very tricky issue.
HTH
Stefan
Re: [Nutch-general] Re: RDF plugin questions
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Stefan - thanks for the reply. I'm still digesting Nutch and how to
work with it at a basic level but it does make sense to allow
metadata to tag along with fetches - I certainly don't know enough
yet to say whether your patch fits into the long-term vision of Nutch
or not yet.
I've started writing a custom RDF parser plugin that will take the
URL and simply add it to Kowari (letting Kowari actually parse it and
ingest it). But I'm feeling like this might not be the best approach.
At what stage would make the most sense for ingesting RDF into an
external system? Is parsing the most logical stage?
Further on this topic, I'm curious about indexing multiple
"documents" per .rdf file fetched - for instance, one document per
RDF "resource". Is this currently possible with a plugin? If not,
what would it take to do something like this? Maybe this approach
doesn't even make sense in the Nutch sense - I'm just exploring my
architectural options.
Thanks,
Erik
On Jul 19, 2005, at 9:19 AM, Stefan Groschupf wrote:
> Hi Erik,
>
> as far I know you have not the source page a url comes from until
> crawl, parse and index time.
> However these information can be extracted from the link graph
> stored in the web db.
> Please have a look to my patch that allows to store custom meta
> data in the web db that can be set until parsing a page and is
> available until indexing time and is merged back to the web db as
> well.
> http://issues.apache.org/jira/browse/NUTCH-59
>
> I still think the patch makes a lot sense and I'm using it for my
> projects in some cases successfully, however it was not accepted
> and committed yet.
> You can find a comment from Doug in the mailing list about the patch.
> He think that with the new map reduce architecture this would be
> obsolete - that is true, but I think it will need a long time
> people will start using map reduce.
> I guess most people using nutch using it for small size special
> interested search engines and mostly integrate them within custom
> applications. The new map reduce architecture will make this
> integration task much much more difficult as it already is.
>
> Greetings,
> Stefan
>
>
> Am 19.07.2005 um 14:57 schrieb Erik Hatcher:
>
>
>> Hi,
>>
>> I'm embarking on an adventure with Nutch to crawl 19th century
>> digital scholarly archives (like the Rossetti Archive, where I
>> work) for the nines.org system. The goal is to use a normal crawl
>> on a selected set of sites, and extract some additional
>> information in the process. Many HTML pages of these archives
>> will be tagged with a <head> tag like this page http://
>> www.rossettiarchive.org/docs/1-1882.s241.raw.html:
>>
>> <link type="application/rdf+xml" title="The Question (for a
>> Design)" href="http://www.rossettiarchive.org/docs/
>> 1-1882.s241.raw.rdf">
>>
>> What I need is a facility to fetch and parse that RDF in a custom
>> way, such that the RDF gets dropped into an RDF engine (currently
>> using Kowari). At the point of processing the RDF data I want to
>> know the URL of the page it came from (the one containing the
>> <link>) such that I can add another RDF statement to the data with
>> provenance information.
>>
>> I see that Nutch fetches the RDF (in the crawl log using the basic
>> crawl command). Can a parse plugin know what page the RDF link
>> came from? If not, then how should I craft things to get that info?
>>
>> At this point I'm a newbie with Nutch, and glad to have this
>> mailing list for advice. I'm quite open to suggestions on how to
>> go about building this custom add-on to Nutch and quite willing to
>> generalize it and contribute it.
>>
>> Thanks,
>> Erik
>>
>>
>>
>>
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> _______________________________________________
> Nutch-general mailing list
> Nutch-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-general
>
Re: RDF plugin questions
Posted by Stefan Groschupf <sg...@media-style.com>.
Hi Erik,
as far I know you have not the source page a url comes from until
crawl, parse and index time.
However these information can be extracted from the link graph stored
in the web db.
Please have a look to my patch that allows to store custom meta data
in the web db that can be set until parsing a page and is available
until indexing time and is merged back to the web db as well.
http://issues.apache.org/jira/browse/NUTCH-59
I still think the patch makes a lot sense and I'm using it for my
projects in some cases successfully, however it was not accepted and
committed yet.
You can find a comment from Doug in the mailing list about the patch.
He think that with the new map reduce architecture this would be
obsolete - that is true, but I think it will need a long time people
will start using map reduce.
I guess most people using nutch using it for small size special
interested search engines and mostly integrate them within custom
applications. The new map reduce architecture will make this
integration task much much more difficult as it already is.
Greetings,
Stefan
Am 19.07.2005 um 14:57 schrieb Erik Hatcher:
> Hi,
>
> I'm embarking on an adventure with Nutch to crawl 19th century
> digital scholarly archives (like the Rossetti Archive, where I
> work) for the nines.org system. The goal is to use a normal crawl
> on a selected set of sites, and extract some additional information
> in the process. Many HTML pages of these archives will be tagged
> with a <head> tag like this page http://www.rossettiarchive.org/
> docs/1-1882.s241.raw.html:
>
> <link type="application/rdf+xml" title="The Question (for a
> Design)" href="http://www.rossettiarchive.org/docs/
> 1-1882.s241.raw.rdf">
>
> What I need is a facility to fetch and parse that RDF in a custom
> way, such that the RDF gets dropped into an RDF engine (currently
> using Kowari). At the point of processing the RDF data I want to
> know the URL of the page it came from (the one containing the
> <link>) such that I can add another RDF statement to the data with
> provenance information.
>
> I see that Nutch fetches the RDF (in the crawl log using the basic
> crawl command). Can a parse plugin know what page the RDF link
> came from? If not, then how should I craft things to get that info?
>
> At this point I'm a newbie with Nutch, and glad to have this
> mailing list for advice. I'm quite open to suggestions on how to
> go about building this custom add-on to Nutch and quite willing to
> generalize it and contribute it.
>
> Thanks,
> Erik
>
>
>