You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Michael Kazekin <Mi...@mediainsight.info> on 2012/02/06 14:10:34 UTC

RSS parser

Hi!

I'm trying to use /plugins/feed/ parser to parse RSS, but have some 
problems:

2012-02-06 15:50:45,340 WARN  parse.ParserFactory - 
ParserFactory:Plugin: org.apache.nutch.parse.feed.FeedParser mapped to 
contentType application/rss+xml via parse-plugins.xml, but its 
plugin.xml file does not claim to support contentType: application/rss+xml
2012-02-06 15:50:45,384 WARN  parse.ParseUtil - Unable to successfully 
parse content http://rss.sciam.com/sciam/earth-and-environment of type 
application/rss+xml
2012-02-06 15:50:45,386 INFO  parse.ParseSegment - Parsing: 
http://rss.sciam.com/sciam/earth-and-environment
2012-02-06 15:50:45,388 WARN  parse.ParseSegment - Error parsing: 
http://rss.sciam.com/sciam/earth-and-environment: failed(2,200): 
org.apache.nutch.parse.ParseException: Unable to successfully parse content


My parse-plugins.xml has:

<mimeType name="application/rss+xml">
<plugin id="feed" />
<!--<plugin id="parse-tika" />-->
</mimeType>

section, and I have

<implementation id="org.apache.nutch.parse.feed.FeedParser" 
class="org.apache.nutch.parse.feed.FeedParser">
<parameter name="contentType" value="application/rss+xml" />
<parameter name="contentType" value="application/atom+xml" />
<parameter name="contentType" value="text/xml" />
<parameter name="pathSuffix" value="rss" />
</implementation>

in /plugins/feed/plugin.xml, and

<property>
<name>mime.type.magic</name>
<value>false</value>
<description>Defines if the mime content type detector uses magic 
resolution.
</description>
</property>

in nutch-default.xml and nutch-site.xml. These also do NOT contain 
"parse-tika" in their plugin.includes property

Also,

2012-02-06 15:50:45,324 INFO  plugin.PluginRepository -     Feed 
Parse/Index/Query Plug-in (feed)

loads normally.



I know about some the issue 
https://issues.apache.org/jira/browse/NUTCH-1053 with TIKA parser, but I 
can't wait until 1.5 is released.

Could you help me with that?


I just need to save RSS links to database (possibly attaching some 
metadata on filtering stage), not choke on them :)





Re: RSS parser

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,

On Tue, Feb 7, 2012 at 3:07 PM, dspathis <ds...@gmail.com> wrote:

>
> In Nutch 1.4, it seems that multiple "contentType" parameters in plugin.xml
> are not handled.
>
> You can merge them into a single parameter. In plugins/feed/plugin.xml, try
> changing
>
>  <implementation id="org.apache.nutch.parse.feed.FeedParser"
> class="org.apache.nutch.parse.feed.FeedParser">
> <parameter name="contentType"
> value="application/rss+xml|application/atom+xml|text/xml" />
> <parameter name="pathSuffix" value="rss" />
> </implementation>
>
> This should get rid of the error "ParserFactory:Plugin:
> org.apache.nutch.parse.feed.FeedParser mapped to
> contentType application/rss+xml via parse-plugins.xml, but its plugin.xml
> file does not claim to support contentType: application/rss+xml" you are
> seeing.
>

Is there any formal explanation for this behaviour? This seems strange...
but if it is a work around then I think we should take notice.
Thanks



-- 
*Lewis*

Re: RSS parser

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,

On Mon, Jul 23, 2012 at 5:00 PM, ShlomiJ <sh...@gmail.com> wrote:
> @Lewis
> Any update on the matter of the differences between Nutch 1.4 and 1.5 ?

No difference between 1.4 & 1.5.1

Lewis

Re: RSS parser

Posted by ShlomiJ <sh...@gmail.com>.
@Lewis
Any update on the matter of the differences between Nutch 1.4 and 1.5 ?

@all
Any new insight on the question why the parse fails?

ShlomiJ



--
View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3996772.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: RSS parser

Posted by Michael Kazekin <Mi...@mediainsight.info>.
Sebastian,

You are right, that was one of the issues I successfully fixed, but 
anyway thank you!


On 05/24/2012 11:28 PM, Sebastian Nagel wrote:
> (it's too late I know)
>
> Have you checked the property http.content.limit
> (default is only 64kB, RSS feeds are often larger).
> Looks like the content is truncated:
>
> > Caused by: com.sun.syndication.io.ParsingFeedException:
> > Invalid XML: Error on line 300: XML document
> > structures must start and end within the same entity.
>
> On 02/10/2012 01:24 PM, Michael Kazekin wrote:
>> On 02/08/2012 06:44 PM, dspathis wrote:
>>>> http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Frss.sciam.com%2Fsciam%2Fearth-and-environment 
>>>>
>>>>
>>> Hmmm. I just tried the URL you provided with my own Nutch 1.4 
>>> installation.
>>> It gets parsed successfully *both* with the feed and with the tika 
>>> parser (I
>>> modified my config first to use the former, then to use the latter).
>>>
>>> I think your config might still have issues. Maybe you could turn on 
>>> TRACE
>>> level logging to see if you can get some clues that way?
>>
>> 1) I installed Nutch 1.4 from scratch,
>>
>> 2) changed nutch-site.xml from empty to:
>>
>> <configuration>
>>
>> <property>
>> <name>http.agent.name</name>
>> <value>Test Nutch Agent (http://www.nutch.org/docs/en/bot.html)</value>
>> </property>
>>
>> <property>
>> <name>http.robots.agents</name>
>> <value>Test Nutch Agent 
>> (http://www.nutch.org/docs/en/bot.html),*</value>
>> </property>
>>
>> </configuration>
>>
>> 3) commented out feed plugin (inparse-plugins.xml)
>>
>> <mimeType name="application/rss+xml">
>> <plugin id="parse-tika" />
>> <!--<plugin id="feed" />-->
>> </mimeType>
>>
>> 4) Changed log level in log4j.properties
>>
>> log4j.logger.org.apache.nutch.fetcher.Fetcher=TRACE,cmdstdout
>> log4j.logger.org.apache.nutch.parse.ParseSegment=TRACE,cmdstdout
>>
>>
>> Then I injected, generated and fetched db with the only RSS link, and 
>> got this exception with Tika:
>>
>>
>> 2012-02-10 15:06:32,782 ERROR tika.TikaParser - Error parsing
>> http://rss.sciam.com/sciam/earth-and-environment
>> org.apache.tika.exception.TikaException: RSS parse error
>> at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:106)
>> at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:95)
>> at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:35)
>> at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:24)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: com.sun.syndication.io.ParsingFeedException: Invalid XML: 
>> Error on line 300: XML document
>> structures must start and end within the same entity.
>> at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:207)
>> at com.sun.syndication.io.SyndFeedInput.build(SyndFeedInput.java:135)
>> at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:68)
>> ... 6 more
>> Caused by: org.jdom.input.JDOMParseException: Error on line 300: XML 
>> document structures must start
>> and end within the same entity.
>> at org.jdom.input.SAXBuilder.build(SAXBuilder.java:468)
>> at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:203)
>> ... 8 more
>> Caused by: org.xml.sax.SAXParseException: XML document structures 
>> must start and end within the same
>> entity.
>> at 
>> org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown 
>> Source)
>> at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
>> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>> at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
>> at 
>> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown 
>> Source)
>> at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown 
>> Source)
>> at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
>> at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
>> at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
>> at 
>> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown 
>> Source)
>> at 
>> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
>> Source)
>> at 
>> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown 
>> Source)
>> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
>> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
>> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
>> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
>> at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown 
>> Source)
>> at org.jdom.input.SAXBuilder.build(SAXBuilder.java:453)
>> ... 9 more
>> 2012-02-10 15:06:32,785 INFO parse.ParseSegment - Parsing:
>> http://rss.sciam.com/sciam/earth-and-environment
>> 2012-02-10 15:06:32,786 WARN parse.ParseSegment - Error parsing:
>> http://rss.sciam.com/sciam/earth-and-environment: failed(2,0): RSS 
>> parse error
>> 2012-02-10 15:06:32,787 INFO crawl.SignatureFactory - Using Signature 
>> impl:
>> org.apache.nutch.crawl.MD5Signature
>> 2012-02-10 15:06:33,235 INFO parse.ParseSegment - ParseSegment: 
>> finished at 2012-02-10 15:06:33,
>> elapsed: 00:00:01
>>
>>> -- 
>>> View this message in context: 
>>> http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3726154.html
>>> Sent from the Nutch - User mailing list archive at Nabble.com.
>>>
>>
>
>
>


Re: RSS parser

Posted by Sebastian Nagel <wa...@googlemail.com>.
(it's too late I know)

Have you checked the property http.content.limit
(default is only 64kB, RSS feeds are often larger).
Looks like the content is truncated:

 > Caused by: com.sun.syndication.io.ParsingFeedException:
 > Invalid XML: Error on line 300: XML document
 > structures must start and end within the same entity.

On 02/10/2012 01:24 PM, Michael Kazekin wrote:
> On 02/08/2012 06:44 PM, dspathis wrote:
>>> http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Frss.sciam.com%2Fsciam%2Fearth-and-environment
>>>
>> Hmmm. I just tried the URL you provided with my own Nutch 1.4 installation.
>> It gets parsed successfully *both* with the feed and with the tika parser (I
>> modified my config first to use the former, then to use the latter).
>>
>> I think your config might still have issues. Maybe you could turn on TRACE
>> level logging to see if you can get some clues that way?
>
> 1) I installed Nutch 1.4 from scratch,
>
> 2) changed nutch-site.xml from empty to:
>
> <configuration>
>
> <property>
> <name>http.agent.name</name>
> <value>Test Nutch Agent (http://www.nutch.org/docs/en/bot.html)</value>
> </property>
>
> <property>
> <name>http.robots.agents</name>
> <value>Test Nutch Agent (http://www.nutch.org/docs/en/bot.html),*</value>
> </property>
>
> </configuration>
>
> 3) commented out feed plugin (inparse-plugins.xml)
>
> <mimeType name="application/rss+xml">
> <plugin id="parse-tika" />
> <!--<plugin id="feed" />-->
> </mimeType>
>
> 4) Changed log level in log4j.properties
>
> log4j.logger.org.apache.nutch.fetcher.Fetcher=TRACE,cmdstdout
> log4j.logger.org.apache.nutch.parse.ParseSegment=TRACE,cmdstdout
>
>
> Then I injected, generated and fetched db with the only RSS link, and got this exception with Tika:
>
>
> 2012-02-10 15:06:32,782 ERROR tika.TikaParser - Error parsing
> http://rss.sciam.com/sciam/earth-and-environment
> org.apache.tika.exception.TikaException: RSS parse error
> at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:106)
> at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:95)
> at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:35)
> at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:24)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.sun.syndication.io.ParsingFeedException: Invalid XML: Error on line 300: XML document
> structures must start and end within the same entity.
> at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:207)
> at com.sun.syndication.io.SyndFeedInput.build(SyndFeedInput.java:135)
> at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:68)
> ... 6 more
> Caused by: org.jdom.input.JDOMParseException: Error on line 300: XML document structures must start
> and end within the same entity.
> at org.jdom.input.SAXBuilder.build(SAXBuilder.java:468)
> at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:203)
> ... 8 more
> Caused by: org.xml.sax.SAXParseException: XML document structures must start and end within the same
> entity.
> at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
> at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
> at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
> at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
> at org.jdom.input.SAXBuilder.build(SAXBuilder.java:453)
> ... 9 more
> 2012-02-10 15:06:32,785 INFO parse.ParseSegment - Parsing:
> http://rss.sciam.com/sciam/earth-and-environment
> 2012-02-10 15:06:32,786 WARN parse.ParseSegment - Error parsing:
> http://rss.sciam.com/sciam/earth-and-environment: failed(2,0): RSS parse error
> 2012-02-10 15:06:32,787 INFO crawl.SignatureFactory - Using Signature impl:
> org.apache.nutch.crawl.MD5Signature
> 2012-02-10 15:06:33,235 INFO parse.ParseSegment - ParseSegment: finished at 2012-02-10 15:06:33,
> elapsed: 00:00:01
>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3726154.html
>> Sent from the Nutch - User mailing list archive at Nabble.com.
>>
>


Re: RSS parser

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Michael,

Please see my output using nutch trunk 1.5. This is parsed using Tika as
I've commented out feed in parse-plugins.xml. Can you please verify with
parsecheker tool and report back. Maybe this is some difference between 1.4
& 1.5?

lewismc@lewismc-HP-Mini-110-3100:~/ASF/trunk/runtime/local$ bin/nutch
parsechecker http://rss.sciam.com/sciam/earth-and-environment
fetching: http://rss.sciam.com/sciam/earth-and-environment
parsing: http://rss.sciam.com/sciam/earth-and-environment
contentType: application/rss+xml
signature: 30ab545d0adff0a08e67d862d3638f79
---------
Url
---------------
http://rss.sciam.com/sciam/earth-and-environment
---------
ParseData
---------
Version: 5
Status: success(1,0)
Title: Scientific American - Environment
Outlinks: 20
  outlink: toUrl:
http://rss.sciam.com/click.phdo?i=7e6d6614015fd2c16cf8ccba2b5516fa anchor:
Is Cadmium as Dangerous for Children as Lead?
  outlink: toUrl:
http://rss.sciam.com/click.phdo?i=305bf340eadb8722162b452b4c52af1c anchor:
Volunteers Plug Holes in the Climate Record (preview)
  outlink: toUrl:
http://rss.sciam.com/click.phdo?i=d3439ec4a1b9646f4961bdfa3766f3f3 anchor:
The Evolving Truth about Fracking for Natural Gas [Updated]
  outlink: toUrl:
http://rss.sciam.com/click.phdo?i=831d81fb0913ab8e440869af44826a00 anchor:
Popular Opinion on Climate Change Traced to Political Elites
  outlink: toUrl:
http://rss.sciam.com/click.phdo?i=660cf40eae46fbc306fe80e9ec8dcf59
...
tons more anchor & outlink data...

Re: RSS parser

Posted by Michael Kazekin <Mi...@mediainsight.info>.
On 02/08/2012 06:44 PM, dspathis wrote:
>> http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Frss.sciam.com%2Fsciam%2Fearth-and-environment
> Hmmm. I just tried the URL you provided with my own Nutch 1.4 installation.
> It gets parsed successfully *both* with the feed and with the tika parser (I
> modified my config first to use the former, then to use the latter).
>
> I think your config might still have issues. Maybe you could turn on TRACE
> level logging to see if you can get some clues that way?

1) I installed Nutch 1.4 from scratch,

2) changed nutch-site.xml from empty to:

<configuration>

<property>
<name>http.agent.name</name>
<value>Test Nutch Agent (http://www.nutch.org/docs/en/bot.html)</value>
</property>

<property>
<name>http.robots.agents</name>
<value>Test Nutch Agent (http://www.nutch.org/docs/en/bot.html),*</value>
</property>

</configuration>

3) commented out feed plugin (inparse-plugins.xml)

<mimeType name="application/rss+xml">
<plugin id="parse-tika" />
<!--<plugin id="feed" />-->
</mimeType>

4) Changed log level in log4j.properties

log4j.logger.org.apache.nutch.fetcher.Fetcher=TRACE,cmdstdout
log4j.logger.org.apache.nutch.parse.ParseSegment=TRACE,cmdstdout


Then I injected, generated and fetched db with the only RSS link, and 
got this exception with Tika:


2012-02-10 15:06:32,782 ERROR tika.TikaParser - Error parsing 
http://rss.sciam.com/sciam/earth-and-environment
org.apache.tika.exception.TikaException: RSS parse error
     at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:106)
     at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:95)
     at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:35)
     at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:24)
     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     at java.lang.Thread.run(Thread.java:662)
Caused by: com.sun.syndication.io.ParsingFeedException: Invalid XML: 
Error on line 300: XML document structures must start and end within the 
same entity.
     at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:207)
     at com.sun.syndication.io.SyndFeedInput.build(SyndFeedInput.java:135)
     at org.apache.tika.parser.feed.FeedParser.parse(FeedParser.java:68)
     ... 6 more
Caused by: org.jdom.input.JDOMParseException: Error on line 300: XML 
document structures must start and end within the same entity.
     at org.jdom.input.SAXBuilder.build(SAXBuilder.java:468)
     at com.sun.syndication.io.WireFeedInput.build(WireFeedInput.java:203)
     ... 8 more
Caused by: org.xml.sax.SAXParseException: XML document structures must 
start and end within the same entity.
     at 
org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown 
Source)
     at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown 
Source)
     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
     at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
     at 
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown 
Source)
     at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown 
Source)
     at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
     at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
     at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
     at 
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source)
     at 
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown 
Source)
     at 
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown 
Source)
     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
     at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
     at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
     at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown 
Source)
     at org.jdom.input.SAXBuilder.build(SAXBuilder.java:453)
     ... 9 more
2012-02-10 15:06:32,785 INFO  parse.ParseSegment - Parsing: 
http://rss.sciam.com/sciam/earth-and-environment
2012-02-10 15:06:32,786 WARN  parse.ParseSegment - Error parsing: 
http://rss.sciam.com/sciam/earth-and-environment: failed(2,0): RSS parse 
error
2012-02-10 15:06:32,787 INFO  crawl.SignatureFactory - Using Signature 
impl: org.apache.nutch.crawl.MD5Signature
2012-02-10 15:06:33,235 INFO  parse.ParseSegment - ParseSegment: 
finished at 2012-02-10 15:06:33, elapsed: 00:00:01

> --
> View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3726154.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>


Re: RSS parser

Posted by dspathis <ds...@gmail.com>.
>http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Frss.sciam.com%2Fsciam%2Fearth-and-environment

Hmmm. I just tried the URL you provided with my own Nutch 1.4 installation.
It gets parsed successfully *both* with the feed and with the tika parser (I
modified my config first to use the former, then to use the latter).

I think your config might still have issues. Maybe you could turn on TRACE
level logging to see if you can get some clues that way?



--
View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3726154.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: RSS parser

Posted by Lewis John Mcgibbney <le...@gmail.com>.
That would be great Michael. Thank you

Lewis

On Tue, Feb 14, 2012 at 4:31 PM, Michael Kazekin <
Michael.Kazekin@mediainsight.info> wrote:

> I'll try to do this and also check if Feed plugin doesn't fail on
> NUTCH-1053 seed with new settings (thus closing the issue).
>
> On 02/08/2012 02:54 PM, Lewis John Mcgibbney wrote:
>
>> Hi,
>>
>> On Wed, Feb 8, 2012 at 8:44 AM, Michael Kazekin<
>> Michael.Kazekin@mediainsight.**info <Mi...@mediainsight.info>>
>>  wrote:
>>
>>  I tried your solution and got rid of "doesn't claim to support
>>> contentType" error indeed.
>>>
>>>  Maybe we should submit a patch for this indeed? Is it possible for you
>> to
>> do this please?
>>
>>
>>
>>  2012-02-07 19:21:48,094 WARN  parse.ParseUtil - Unable to successfully
>>> parse content http://rss.sciam.com/sciam/**earth-and-environment<http://rss.sciam.com/sciam/earth-and-environment>of type
>>> application/rss+xml
>>>
>>>
>>>  There is an issue for the feed plugin [1], can you please have a look
>> through and see if any of this looks familiar.
>> Thank you
>>
>> [1] https://issues.apache.org/**jira/browse/NUTCH-1053<https://issues.apache.org/jira/browse/NUTCH-1053>
>>
>>
>


-- 
*Lewis*

Re: RSS parser

Posted by Michael Kazekin <Mi...@mediainsight.info>.
I'll try to do this and also check if Feed plugin doesn't fail on 
NUTCH-1053 seed with new settings (thus closing the issue).

On 02/08/2012 02:54 PM, Lewis John Mcgibbney wrote:
> Hi,
>
> On Wed, Feb 8, 2012 at 8:44 AM, Michael Kazekin<
> Michael.Kazekin@mediainsight.info>  wrote:
>
>> I tried your solution and got rid of "doesn't claim to support
>> contentType" error indeed.
>>
> Maybe we should submit a patch for this indeed? Is it possible for you to
> do this please?
>
>
>
>> 2012-02-07 19:21:48,094 WARN  parse.ParseUtil - Unable to successfully
>> parse content http://rss.sciam.com/sciam/earth-and-environment of type
>> application/rss+xml
>>
>>
> There is an issue for the feed plugin [1], can you please have a look
> through and see if any of this looks familiar.
> Thank you
>
> [1] https://issues.apache.org/jira/browse/NUTCH-1053
>


Re: RSS parser

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,

On Wed, Feb 8, 2012 at 8:44 AM, Michael Kazekin <
Michael.Kazekin@mediainsight.info> wrote:

>
> I tried your solution and got rid of "doesn't claim to support
> contentType" error indeed.
>

Maybe we should submit a patch for this indeed? Is it possible for you to
do this please?



> 2012-02-07 19:21:48,094 WARN  parse.ParseUtil - Unable to successfully
> parse content http://rss.sciam.com/sciam/earth-and-environment of type
> application/rss+xml
>
>
There is an issue for the feed plugin [1], can you please have a look
through and see if any of this looks familiar.
Thank you

[1] https://issues.apache.org/jira/browse/NUTCH-1053

Re: RSS parser

Posted by Michael Kazekin <Mi...@mediainsight.info>.
Thank you for your help!

I tried your solution and got rid of "doesn't claim to support 
contentType" error indeed.

But still have

2012-02-07 19:21:48,094 WARN  parse.ParseUtil - Unable to successfully 
parse content http://rss.sciam.com/sciam/earth-and-environment of type 
application/rss+xml
2012-02-07 19:21:48,097 INFO  parse.ParseSegment - Parsing: 
http://rss.sciam.com/sciam/earth-and-environment
2012-02-07 19:21:48,098 WARN  parse.ParseSegment - Error parsing: 
http://rss.sciam.com/sciam/earth-and-environment: failed(2,200): 
org.apache.nutch.parse.ParseException: Unable to successfully parse content

I checked my link with W3C RSS validator, and it passed:

http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Frss.sciam.com%2Fsciam%2Fearth-and-environment

For now I just need to differentiate RSS links from all others by 
guessing their content, so my needs are simpler
than abilities of average parser plugin. But I still don't know, whether 
to subclass feed/Tika plugin or write my own.


On 02/07/2012 07:07 PM, dspathis wrote:
> Hi,
>
> In Nutch 1.4, it seems that multiple "contentType" parameters in plugin.xml
> are not handled.
>
> You can merge them into a single parameter. In plugins/feed/plugin.xml, try
> changing
>
> <implementation id="org.apache.nutch.parse.feed.FeedParser"
> class="org.apache.nutch.parse.feed.FeedParser">
> <parameter name="contentType" value="application/rss+xml" />
> <parameter name="contentType" value="application/atom+xml" />
> <parameter name="contentType" value="text/xml" />
> <parameter name="pathSuffix" value="rss" />
> </implementation>
>
> to
>
> <implementation id="org.apache.nutch.parse.feed.FeedParser"
> class="org.apache.nutch.parse.feed.FeedParser">
> <parameter name="contentType"
> value="application/rss+xml|application/atom+xml|text/xml" />
> <parameter name="pathSuffix" value="rss" />
> </implementation>
>
> This should get rid of the error "ParserFactory:Plugin:
> org.apache.nutch.parse.feed.FeedParser mapped to
> contentType application/rss+xml via parse-plugins.xml, but its plugin.xml
> file does not claim to support contentType: application/rss+xml" you are
> seeing.
>
> However, you may still have issues parsing RSS, even with the feed plugin.
> It's possibly due to malformed RSS. Have you tried running the RSS in
> question through a validator?
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3722778.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>


Re: RSS parser

Posted by dspathis <ds...@gmail.com>.
Hi,

In Nutch 1.4, it seems that multiple "contentType" parameters in plugin.xml
are not handled.

You can merge them into a single parameter. In plugins/feed/plugin.xml, try
changing

<implementation id="org.apache.nutch.parse.feed.FeedParser"
class="org.apache.nutch.parse.feed.FeedParser">
<parameter name="contentType" value="application/rss+xml" />
<parameter name="contentType" value="application/atom+xml" />
<parameter name="contentType" value="text/xml" />
<parameter name="pathSuffix" value="rss" />
</implementation> 

to

<implementation id="org.apache.nutch.parse.feed.FeedParser"
class="org.apache.nutch.parse.feed.FeedParser">
<parameter name="contentType"
value="application/rss+xml|application/atom+xml|text/xml" />
<parameter name="pathSuffix" value="rss" />
</implementation> 

This should get rid of the error "ParserFactory:Plugin:
org.apache.nutch.parse.feed.FeedParser mapped to
contentType application/rss+xml via parse-plugins.xml, but its plugin.xml
file does not claim to support contentType: application/rss+xml" you are
seeing.

However, you may still have issues parsing RSS, even with the feed plugin.
It's possibly due to malformed RSS. Have you tried running the RSS in
question through a validator?


--
View this message in context: http://lucene.472066.n3.nabble.com/RSS-parser-tp3719558p3722778.html
Sent from the Nutch - User mailing list archive at Nabble.com.