You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ar...@csiro.au on 2015/06/23 03:59:55 UTC
A parser failure on a single document may fail crawling job
Hi,
This is what happened:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:213)
<...>
Caused by: java.lang.IncompatibleClassChangeError: class org.apache.tika.parser.asm.XHTMLClassVisitor has interface org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.tika.parser.asm.ClassParser.parse(ClassParser.java:51)
at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:98)
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:103)
Suggested fix in ParseUtil:
Replace
if (maxParseTime!=-1)
parseResult = runParser(parsers[i], content);
else
parseResult = parsers[i].getParse(content);
with
try
{
if (maxParseTime!=-1)
parseResult = runParser(parsers[i], content);
else
parseResult = parsers[i].getParse(content);
} catch( Throwable e )
{
LOG.warn( "Parsing " + content.getUrl() + " with " + parsers[i].getClass().getName() + " failed: " + e.getMessage() ) ;
parseResult = null ;
}
Also replace
if (maxParseTime!=-1)
parseResult = runParser(p, content);
else
parseResult = p.getParse(content);
with
try
{
if (maxParseTime!=-1)
parseResult = runParser(p, content);
else
parseResult = p.getParse(content);
} catch( Throwable e )
{
LOG.warn( "Parsing " + content.getUrl() + " with " + p.getClass().getName() + " failed: " + e.getMessage() ) ;
}
Regards,
Arkadi
RE: A parser failure on a single document may fail crawling job
Posted by Ar...@csiro.au.
Hi Sebastian,
I apologise for a long silence on this issue. I have been out of town, back on Monday. Then I will do what you are asking in 2-3 days.
Regards,
Arkadi
________________________________________
From: Sebastian Nagel [wastl.nagel@googlemail.com]
Sent: Friday, 24 July 2015 6:38 AM
To: user@nutch.apache.org
Cc: Kosmynin, Arkadi (CASS, Marsfield)
Subject: Re: A parser failure on a single document may fail crawling job
Hi Arkadi,
does the problem persist?
Which version of Nutch are you using?
Can you point to one file or URL to reproduce it?
Thanks,
Sebastian
On 06/26/2015 03:26 PM, Sebastian Nagel wrote:
> Hi Arkadi,
>
> thanks for reporting that. Can you open a Jira ticket [1] to address this bug?
>
> It's rather a bug of the plugin parse-tika and should be solved there,
> cf. https://issues.apache.org/jira/browse/TIKA-1240
> A plugin should be able to load all required classes.
>
> Thanks,
> Sebastian
>
> [1] https://issues.apache.org/jira/browse/NUTCH
>
> 2015-06-23 3:59 GMT+02:00 <Arkadi.Kosmynin@csiro.au <ma...@csiro.au>>:
>
> Hi,
>
> This is what happened:
>
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
> at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:213)
> <...>
> Caused by: java.lang.IncompatibleClassChangeError: class
> org.apache.tika.parser.asm.XHTMLClassVisitor has interface org.objectweb.asm.ClassVisitor as
> super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.tika.parser.asm.ClassParser.parse(ClassParser.java:51)
> at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:98)
> at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:103)
>
> Suggested fix in ParseUtil:
>
> Replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " + parsers[i].getClass().getName() + "
> failed: " + e.getMessage() ) ;
> parseResult = null ;
> }
>
> Also replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " + p.getClass().getName() + " failed: "
> + e.getMessage() ) ;
> }
>
> Regards,
> Arkadi
>
>
RE: A parser failure on a single document may fail crawling job
Posted by Ar...@csiro.au.
Hi Sebastian,
> -----Original Message-----
> From: Sebastian Nagel [mailto:wastl.nagel@googlemail.com]
> Sent: Friday, 24 July 2015 6:39 AM
> To: user@nutch.apache.org
> Cc: Kosmynin, Arkadi (CASS, Marsfield) <Ar...@csiro.au>
> Subject: Re: A parser failure on a single document may fail crawling job
>
> Hi Arkadi,
>
> does the problem persist?
Yes.
> Which version of Nutch are you using?
1.9
> Can you point to one file or URL to reproduce it?
To reproduce:
- Remove a jar file that one of your parsers depends on.
- Make Nutch parse any file using this parser.
This will result in NoSuchMethodError thrown and crawling job failed.
I've created a JIRA issue NUTCH-2071 and attached a patch. I believe that this problem should be handled at ParseUtil level because people may use their own or third party parsers and Nutch should be protected from parsers problems.
Regards,
Arkadi
>
> Thanks,
> Sebastian
>
> On 06/26/2015 03:26 PM, Sebastian Nagel wrote:
> > Hi Arkadi,
> >
> > thanks for reporting that. Can you open a Jira ticket [1] to address this bug?
> >
> > It's rather a bug of the plugin parse-tika and should be solved there,
> > cf. https://issues.apache.org/jira/browse/TIKA-1240
> > A plugin should be able to load all required classes.
> >
> > Thanks,
> > Sebastian
> >
> > [1] https://issues.apache.org/jira/browse/NUTCH
> >
> > 2015-06-23 3:59 GMT+02:00 <Arkadi.Kosmynin@csiro.au
> <ma...@csiro.au>>:
> >
> > Hi,
> >
> > This is what happened:
> >
> > java.io.IOException: Job failed!
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
> > at
> org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:213)
> > <...>
> > Caused by: java.lang.IncompatibleClassChangeError: class
> > org.apache.tika.parser.asm.XHTMLClassVisitor has interface
> org.objectweb.asm.ClassVisitor as
> > super class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at
> org.apache.tika.parser.asm.ClassParser.parse(ClassParser.java:51)
> > at
> org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:98)
> > at
> > org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:103)
> >
> > Suggested fix in ParseUtil:
> >
> > Replace
> >
> > if (maxParseTime!=-1)
> > parseResult = runParser(parsers[i], content);
> > else
> > parseResult = parsers[i].getParse(content);
> >
> > with
> >
> > try
> > {
> > if (maxParseTime!=-1)
> > parseResult = runParser(parsers[i], content);
> > else
> > parseResult = parsers[i].getParse(content);
> > } catch( Throwable e )
> > {
> > LOG.warn( "Parsing " + content.getUrl() + " with " +
> parsers[i].getClass().getName() + "
> > failed: " + e.getMessage() ) ;
> > parseResult = null ;
> > }
> >
> > Also replace
> >
> > if (maxParseTime!=-1)
> > parseResult = runParser(p, content);
> > else
> > parseResult = p.getParse(content);
> >
> > with
> >
> > try
> > {
> > if (maxParseTime!=-1)
> > parseResult = runParser(p, content);
> > else
> > parseResult = p.getParse(content);
> > } catch( Throwable e )
> > {
> > LOG.warn( "Parsing " + content.getUrl() + " with " +
> p.getClass().getName() + " failed: "
> > + e.getMessage() ) ;
> > }
> >
> > Regards,
> > Arkadi
> >
> >
Re: A parser failure on a single document may fail crawling job
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Arkadi,
does the problem persist?
Which version of Nutch are you using?
Can you point to one file or URL to reproduce it?
Thanks,
Sebastian
On 06/26/2015 03:26 PM, Sebastian Nagel wrote:
> Hi Arkadi,
>
> thanks for reporting that. Can you open a Jira ticket [1] to address this bug?
>
> It's rather a bug of the plugin parse-tika and should be solved there,
> cf. https://issues.apache.org/jira/browse/TIKA-1240
> A plugin should be able to load all required classes.
>
> Thanks,
> Sebastian
>
> [1] https://issues.apache.org/jira/browse/NUTCH
>
> 2015-06-23 3:59 GMT+02:00 <Arkadi.Kosmynin@csiro.au <ma...@csiro.au>>:
>
> Hi,
>
> This is what happened:
>
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
> at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:213)
> <...>
> Caused by: java.lang.IncompatibleClassChangeError: class
> org.apache.tika.parser.asm.XHTMLClassVisitor has interface org.objectweb.asm.ClassVisitor as
> super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.tika.parser.asm.ClassParser.parse(ClassParser.java:51)
> at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:98)
> at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:103)
>
> Suggested fix in ParseUtil:
>
> Replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " + parsers[i].getClass().getName() + "
> failed: " + e.getMessage() ) ;
> parseResult = null ;
> }
>
> Also replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " + p.getClass().getName() + " failed: "
> + e.getMessage() ) ;
> }
>
> Regards,
> Arkadi
>
>
Re: A parser failure on a single document may fail crawling job
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Arkadi,
thanks for reporting that. Can you open a Jira ticket [1] to address this
bug?
It's rather a bug of the plugin parse-tika and should be solved there,
cf. https://issues.apache.org/jira/browse/TIKA-1240
A plugin should be able to load all required classes.
Thanks,
Sebastian
[1] https://issues.apache.org/jira/browse/NUTCH
2015-06-23 3:59 GMT+02:00 <Ar...@csiro.au>:
> Hi,
>
> This is what happened:
>
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
> at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:213)
> <...>
> Caused by: java.lang.IncompatibleClassChangeError: class
> org.apache.tika.parser.asm.XHTMLClassVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at
> java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.tika.parser.asm.ClassParser.parse(ClassParser.java:51)
> at
> org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:98)
> at
> org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:103)
>
> Suggested fix in ParseUtil:
>
> Replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(parsers[i], content);
> else
> parseResult = parsers[i].getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " +
> parsers[i].getClass().getName() + " failed: " + e.getMessage() ) ;
> parseResult = null ;
> }
>
> Also replace
>
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
>
> with
>
> try
> {
> if (maxParseTime!=-1)
> parseResult = runParser(p, content);
> else
> parseResult = p.getParse(content);
> } catch( Throwable e )
> {
> LOG.warn( "Parsing " + content.getUrl() + " with " +
> p.getClass().getName() + " failed: " + e.getMessage() ) ;
> }
>
> Regards,
> Arkadi
>