Parsing XML files

[Mike Reynols <au...@hotmail.com>]

Is there a plugin of some sort that I need in order to take a web site 
(which serves up a collection of xml documents) and crawl it's non html 
files?

I have tried to crawl an apache server of mine that has a directory listing 
of several hundred xml files but it failed with:

051108 113634 fetching http://www.example.com/example.xml.43779
051108 113640 fetch okay, but can't parse 
http://www.example.com/example.xml.43792, reason: failed(2,203): 
Content-Type not text/html: application/xml
051108 113640 fetching http://buildhost.kozoru.com/example.xml.43812

Now when I stripped out all the xml and left just raw text, I recieved the 
following error:

051108 113634 fetching http://www.example.com/example.xml.43779-txt
051108 113640 fetch okay, but can't parse 
http://www.example.com/example.xml.43792-txt, reason: failed(2,203): 
Content-Type not text/html: application/xml
051108 113640 fetching http://www.example.com/example.xml.43812-txt

So you can see that niether are parsing correctly, and I'm not entirely sure 
why? Is there any way I can parse a collection of non-html files and be able 
to search it?

I guess I'm confused as to the fundamentals of Nutch. If someone could 
please point me in the right direction, that'd be greatly appreciated. 
Thanks.

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Parsing XML files

[Mike Reynols <au...@hotmail.com>]

Thanks for the help bud.

Yes, I changed that name of my files, but that was not the only change that 
was made. All xml nodes/tags were removed from the documents.

I figured out that Nutch would not even attempt to parse these documents 
while 'xml' was in the file name. So I simply renamed them (again), 
excluding the .xml. portion and it parsed just fine.

Again, thanks for the help.

>From: Jérôme Charron <je...@gmail.com>
>Reply-To: nutch-user@lucene.apache.org
>To: nutch-user@lucene.apache.org
>Subject: Re: Parsing XML files
>Date: Tue, 8 Nov 2005 21:33:34 +0100
>
>On 11/8/05, Mike Reynols <au...@hotmail.com> wrote:
> >
> > Is there a plugin of some sort that I need in order to take a web site
> > (which serves up a collection of xml documents) and crawl it's non html
> > files?
>
>Hi Mike,
>
>First of all, which nutch version are you using?
>Concerning a xml, there's actually no parse-xml plugin in nutch.
>We have currently some discussion with two other nutch developpers to
>provide such plugin... but it is still in early stages.
>
>Now when I stripped out all the xml and left just raw text, I recieved the
> > following error:
>
>Ok, you renamed your documents... but what is the mime-type returned by 
>your
>server?
>It seems it is application/xml => there's no parse plugin that handle such
>content-type.
>Regards
>
>Jérôme
>
>--
>http://motrech.free.fr/
>http://www.frutch.org/

_________________________________________________________________
Don’t just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

Re: Parsing XML files

[Jérôme Charron <je...@gmail.com>]

On 11/8/05, Mike Reynols <au...@hotmail.com> wrote:
>
> Is there a plugin of some sort that I need in order to take a web site
> (which serves up a collection of xml documents) and crawl it's non html
> files?

Hi Mike,

First of all, which nutch version are you using?
Concerning a xml, there's actually no parse-xml plugin in nutch.
We have currently some discussion with two other nutch developpers to
provide such plugin... but it is still in early stages.

Now when I stripped out all the xml and left just raw text, I recieved the
> following error:

Ok, you renamed your documents... but what is the mime-type returned by your
server?
It seems it is application/xml => there's no parse plugin that handle such
content-type.
Regards

Jérôme

--
http://motrech.free.fr/
http://www.frutch.org/