You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by devang pandey <de...@gmail.com> on 2013/07/08 13:46:20 UTC
nutch 1.2 solr 3.1 integration issue
I have crawled a site successfully using NUTCH 1.2 .Now I want to integrate
this with solr 3.1 . Problem is when I am issuing command $ bin/nutch
solrindex http://localhost:8080/solr/ crawl/crawldb crawl/linkdb cra
wl/segments/* an error occurs. I am attaching my nutch logs
Please help me to solve this issue
java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpException
at org.apache.nutch.indexer.solr.SolrWriter.open(SolrWriter.java:44)
at
org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:42)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:433)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.httpclient.HttpException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 5 more
2013-07-08 17:11:29,369 ERROR solr.SolrIndexer - java.io.IOException: Job
failed!
Re: nutch 1.2 solr 3.1 integration issue
Posted by devang pandey <de...@gmail.com>.
hey markus able to resolve httpclient issue ...but not having a new issue
org.apache.solr.common.SolrException: Bad Request
Bad Request
request: http://localhost:8080/solr/update?wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:436)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:245)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75)
at
org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
2013-07-08 17:38:47,577 ERROR solr.SolrIndexer - java.io.IOException: Job
failed!
On Mon, Jul 8, 2013 at 5:23 PM, Markus Jelsma <ma...@openindex.io>wrote:
> You're not using the correct httpclient jar for the solrj client version.
> This is some dependency hell. You'd better build 1.2 from source and change
> the versions in ivy. That would be easier i think.
>
> -----Original message-----
> > From:devang pandey <de...@gmail.com>
> > Sent: Monday 8th July 2013 13:46
> > To: user@nutch.apache.org
> > Subject: nutch 1.2 solr 3.1 integration issue
> >
> > I have crawled a site successfully using NUTCH 1.2 .Now I want to
> integrate
> > this with solr 3.1 . Problem is when I am issuing command $ bin/nutch
> > solrindex http://localhost:8080/solr/ crawl/crawldb crawl/linkdb cra
> > wl/segments/* an error occurs. I am attaching my nutch logs
> >
> > Please help me to solve this issue
> > java.lang.NoClassDefFoundError:
> org/apache/commons/httpclient/HttpException
> > at org.apache.nutch.indexer.solr.SolrWriter.open(SolrWriter.java:44)
> > at
> >
> org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:42)
> > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:433)
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> > at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.commons.httpclient.HttpException
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> > ... 5 more
> > 2013-07-08 17:11:29,369 ERROR solr.SolrIndexer - java.io.IOException: Job
> > failed!
> >
>
RE: nutch 1.2 solr 3.1 integration issue
Posted by Markus Jelsma <ma...@openindex.io>.
You're not using the correct httpclient jar for the solrj client version. This is some dependency hell. You'd better build 1.2 from source and change the versions in ivy. That would be easier i think.
-----Original message-----
> From:devang pandey <de...@gmail.com>
> Sent: Monday 8th July 2013 13:46
> To: user@nutch.apache.org
> Subject: nutch 1.2 solr 3.1 integration issue
>
> I have crawled a site successfully using NUTCH 1.2 .Now I want to integrate
> this with solr 3.1 . Problem is when I am issuing command $ bin/nutch
> solrindex http://localhost:8080/solr/ crawl/crawldb crawl/linkdb cra
> wl/segments/* an error occurs. I am attaching my nutch logs
>
> Please help me to solve this issue
> java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpException
> at org.apache.nutch.indexer.solr.SolrWriter.open(SolrWriter.java:44)
> at
> org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:42)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:433)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.httpclient.HttpException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> ... 5 more
> 2013-07-08 17:11:29,369 ERROR solr.SolrIndexer - java.io.IOException: Job
> failed!
>
RE: nutch 1.2 solr 3.1 integration issue
Posted by Markus Jelsma <ma...@openindex.io>.
Ah, you were able to communicate with Solr now. But an error occurred on Solr's side. Check Solr's logs for more information. Likely a field is missing or is not defined as multivalued or some type is wrong.
-----Original message-----
> From:devang pandey <de...@gmail.com>
> Sent: Monday 8th July 2013 14:15
> To: user@nutch.apache.org
> Subject: Re: nutch 1.2 solr 3.1 integration issue
>
> hey markus able to resolve httpclient issue ...but not having a new issue
> org.apache.solr.common.SolrException: Bad Request
>
> Bad Request
>
> request: http://localhost:8080/solr/update?wt=javabin&version=2
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:436)
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:245)
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
> at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75)
> at
> org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> 2013-07-08 17:38:47,577 ERROR solr.SolrIndexer - java.io.IOException: Job
> failed!
>
>
>
> On Mon, Jul 8, 2013 at 5:23 PM, Markus Jelsma <ma...@openindex.io>wrote:
>
> > You're not using the correct httpclient jar for the solrj client version.
> > This is some dependency hell. You'd better build 1.2 from source and change
> > the versions in ivy. That would be easier i think.
> >
> > -----Original message-----
> > > From:devang pandey <de...@gmail.com>
> > > Sent: Monday 8th July 2013 13:46
> > > To: user@nutch.apache.org
> > > Subject: nutch 1.2 solr 3.1 integration issue
> > >
> > > I have crawled a site successfully using NUTCH 1.2 .Now I want to
> > integrate
> > > this with solr 3.1 . Problem is when I am issuing command $ bin/nutch
> > > solrindex http://localhost:8080/solr/ crawl/crawldb crawl/linkdb cra
> > > wl/segments/* an error occurs. I am attaching my nutch logs
> > >
> > > Please help me to solve this issue
> > > java.lang.NoClassDefFoundError:
> > org/apache/commons/httpclient/HttpException
> > > at org.apache.nutch.indexer.solr.SolrWriter.open(SolrWriter.java:44)
> > > at
> > >
> > org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:42)
> > > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:433)
> > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> > > at
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.commons.httpclient.HttpException
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> > > ... 5 more
> > > 2013-07-08 17:11:29,369 ERROR solr.SolrIndexer - java.io.IOException: Job
> > > failed!
> > >
> >
>