You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Max Stricker <ma...@maxstricker.it> on 2011/04/19 14:59:28 UTC
Nutch 1.2 Solr 3.1.0 solrindex Job failed
Hi all,
I am trying to run Nutch and Solr using the tutorial from [1].
When indexing the content to Solr using command
bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb crawl/linkdb crawl/segments/*
I see the following error:
SolrIndexer: starting at 2011-04-19 03:00:52
java.io.IOException: Job failed!
The log indicates a Runtime Error when unmarshalling an object:
2011-04-19 03:01:26,630 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-04-19 03:01:26,630 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-04-19 03:01:26,634 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter
2011-04-19 03:01:26,635 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2011-04-19 03:01:26,673 INFO solr.SolrMappingReader - source: content dest: content
2011-04-19 03:01:26,673 INFO solr.SolrMappingReader - source: site dest: site
2011-04-19 03:01:26,673 INFO solr.SolrMappingReader - source: title dest: title
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: host dest: host
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: segment dest: segment
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: boost dest: boost
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: digest dest: digest
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: url dest: id
2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: url dest: url
2011-04-19 03:01:28,544 WARN mapred.LocalJobRunner - job_local_0001
java.lang.RuntimeException: Invalid version or the data in not in 'javabin' format
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:39)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:466)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75)
at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
2011-04-19 03:01:29,472 ERROR solr.SolrIndexer - java.io.IOException: Job failed!
Any ideas what could go wrong here? I strictly followed the mentioned tutorial.
The index should be fine:
m@ubuntu:~/Desktop/nutch$ bin/nutch readdb crawl/crawldb -stats
CrawlDb statistics start: crawl/crawldb
Statistics for CrawlDb: crawl/crawldb
TOTAL urls: 284
retry 0: 284
min score: 0.0
avg score: 0.030411972
max score: 1.208
status 1 (db_unfetched): 17
status 2 (db_fetched): 227
status 3 (db_gone): 40
CrawlDb statistics: done
regards,
Max
[1] http://wiki.apache.org/nutch/RunningNutchAndSolr
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Max Stricker <ma...@maxstricker.it>.
The script seems not a solution at the moment as the final
system should run on a Windows machine (using cygwin).
The Solr testdrive tutorial seems to not work with my configuration:
m@ubuntu:~/Desktop/solr/example/exampledocs$ java -jar post.jar *.xml
SimplePostTool: version 1.3
SimplePostTool: POSTing files to http://localhost:8983/solr/update..
SimplePostTool: POSTing file gb18030-example.xml
SimplePostTool: FATAL: Solr returned an error #400 ERROR:unknown field 'name'
Are there some parts missing in the tutorial I mentioned first?
Am 19.04.2011 um 15:33 schrieb Gabriele Kahlout:
> Have you tried solr testdrive tutorial[2]?
>
> Also this script[1] worked for me (see outputs) with nutch 1.2 and Solr 3.1
> Try the 3rd script as follows:
> Change $solrIndex from "http://localhost:8080/solr" to "
> http://127.0.0.1:8983/solr/"
> $ ./whole-web-crawling-incremental -f dirWithSeeds
> where dirWithSeeds contains a txt file whole name contains url (i.e. *url).
>
>
>
>
> [1] http://wiki.apache.org/nutch/Whole-Web%20Crawling%20incremental%20script
> [2] http://lucene.apache.org/solr/tutorial.html
> On Tue, Apr 19, 2011 at 3:19 PM, Max Stricker <ma...@maxstricker.it> wrote:
>
>>
>>> You must upgrade Nutch' two SolrJ jar's to 3.1.
>>
>> That's what I already tried. I copied the SolrJ Jars contained in the Solr
>> 3.1.0 release
>> to nutch/lib and removed the old ones.
>> So nutch now uses apache-solr-solrj-3.1.0.jar
>> The error however remains the same.
>>
>> Any other suggestions?
>
>
>
>
> --
> Regards,
> K. Gabriele
>
> --- unchanged since 20/9/10 ---
> P.S. If the subject contains "[LON]" or the addressee acknowledges the
> receipt within 48 hours then I don't resend the email.
> subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
> < Now + 48h) ⇒ ¬resend(I, this).
>
> If an email is sent by a sender that is not a trusted contact or the email
> does not contain a valid code then the email is not received. A valid code
> starts with a hyphen and ends with "X".
> ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
> L(-[a-z]+[0-9]X)).
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Gabriele Kahlout <ga...@mysimpatico.com>.
Have you tried solr testdrive tutorial[2]?
Also this script[1] worked for me (see outputs) with nutch 1.2 and Solr 3.1
Try the 3rd script as follows:
Change $solrIndex from "http://localhost:8080/solr" to "
http://127.0.0.1:8983/solr/"
$ ./whole-web-crawling-incremental -f dirWithSeeds
where dirWithSeeds contains a txt file whole name contains url (i.e. *url).
[1] http://wiki.apache.org/nutch/Whole-Web%20Crawling%20incremental%20script
[2] http://lucene.apache.org/solr/tutorial.html
On Tue, Apr 19, 2011 at 3:19 PM, Max Stricker <ma...@maxstricker.it> wrote:
>
> > You must upgrade Nutch' two SolrJ jar's to 3.1.
>
> That's what I already tried. I copied the SolrJ Jars contained in the Solr
> 3.1.0 release
> to nutch/lib and removed the old ones.
> So nutch now uses apache-solr-solrj-3.1.0.jar
> The error however remains the same.
>
> Any other suggestions?
--
Regards,
K. Gabriele
--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).
If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Markus Jelsma <ma...@openindex.io>.
Your version is wrong. Also make sure you're not using class
solr.BinaryUpdateRequestHandler as in your update RequestHandler, it'll fail
for some reason.
It does work, i just posted from Nutch 1.3-dev to Solr 3.1.
On Tuesday 19 April 2011 21:16:18 Max Stricker wrote:
> So now it gets interesting:
> I copied nutch lib and have done a successful 'ant job'.
> bin/nutch solrindex still raises a java.io.IOException: Job failed!
> The interesting part is in the solr output:
>
> Apr 19, 2011 11:57:31 AM
> org.apache.solr.update.processor.LogUpdateProcessor finish INFO:
> {add=[http:/urltoindex.com/, http:/urltoindex.com/2010/04/12.html, ...
> (162 adds)]} 0 1746 Apr 19, 2011 11:57:31 AM org.apache.solr.core.SolrCore
> execute
> INFO: [] webapp=/solr path=/update params={wt=javabin&version=1} status=0
> QTime=1746
>
> So it seems that nutch passes some information to solr.
> But the nutch log states the following:
> 2011-04-19 12:08:05,221 WARN mapred.LocalJobRunner - job_local_0001
> java.lang.RuntimeException: Invalid version or the data in not in 'javabin'
> format at
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
> at
> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(Bin
> aryResponseParser.java:39) at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHtt
> pSolrServer.java:466) at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHtt
> pSolrServer.java:243) at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(Abstrac
> tUpdateRequest.java:105) at
> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49) at
> org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75) at
> org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.j
> ava:48) at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> 2011-04-19 12:08:06,042 ERROR solr.SolrIndexer - java.io.IOException: Job
> failed!
>
> What could be wrong with that simple setup?
> Could there be other jar incompatibilities?
>
> > Copying to Nutch lib does not necessarily mean that this is what is being
> > used until you generate a new job file with 'ant job'. Nutch 1.3
> > clarifies this by separating a deployed environment from a local one and
> > it would be a good idea to use it (it will be released pretty soon
> > anyway).
> >
> > Am starting to believe that we should upgrade the version of SOLR in 1.3
> > before it is releases as I expect that many people will be using it.
> >
> > On 19 April 2011 14:19, Max Stricker <ma...@maxstricker.it> wrote:
> >>> You must upgrade Nutch' two SolrJ jar's to 3.1.
> >>
> >> That's what I already tried. I copied the SolrJ Jars contained in the
> >> Solr 3.1.0 release
> >> to nutch/lib and removed the old ones.
> >> So nutch now uses apache-solr-solrj-3.1.0.jar
> >> The error however remains the same.
> >>
> >> Any other suggestions?
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Markus Jelsma <ma...@openindex.io>.
> Copying to Nutch lib does not necessarily mean that this is what is being
> used until you generate a new job file with 'ant job'. Nutch 1.3 clarifies
> this by separating a deployed environment from a local one and it would be
> a good idea to use it (it will be released pretty soon anyway).
>
> Am starting to believe that we should upgrade the version of SOLR in 1.3
> before it is releases as I expect that many people will be using it.
Yes:
https://issues.apache.org/jira/browse/NUTCH-983
>
> On 19 April 2011 14:19, Max Stricker <ma...@maxstricker.it> wrote:
> > > You must upgrade Nutch' two SolrJ jar's to 3.1.
> >
> > That's what I already tried. I copied the SolrJ Jars contained in the
> > Solr 3.1.0 release
> > to nutch/lib and removed the old ones.
> > So nutch now uses apache-solr-solrj-3.1.0.jar
> > The error however remains the same.
> >
> > Any other suggestions?
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Max Stricker <ma...@maxstricker.it>.
So now it gets interesting:
I copied nutch lib and have done a successful 'ant job'.
bin/nutch solrindex still raises a java.io.IOException: Job failed!
The interesting part is in the solr output:
Apr 19, 2011 11:57:31 AM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[http:/urltoindex.com/, http:/urltoindex.com/2010/04/12.html, ... (162 adds)]} 0 1746
Apr 19, 2011 11:57:31 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=javabin&version=1} status=0 QTime=1746
So it seems that nutch passes some information to solr.
But the nutch log states the following:
2011-04-19 12:08:05,221 WARN mapred.LocalJobRunner - job_local_0001
java.lang.RuntimeException: Invalid version or the data in not in 'javabin' format
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:39)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:466)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75)
at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
2011-04-19 12:08:06,042 ERROR solr.SolrIndexer - java.io.IOException: Job failed!
What could be wrong with that simple setup?
Could there be other jar incompatibilities?
> Copying to Nutch lib does not necessarily mean that this is what is being
> used until you generate a new job file with 'ant job'. Nutch 1.3 clarifies
> this by separating a deployed environment from a local one and it would be a
> good idea to use it (it will be released pretty soon anyway).
>
> Am starting to believe that we should upgrade the version of SOLR in 1.3
> before it is releases as I expect that many people will be using it.
>
>
> On 19 April 2011 14:19, Max Stricker <ma...@maxstricker.it> wrote:
>
>>
>>> You must upgrade Nutch' two SolrJ jar's to 3.1.
>>
>> That's what I already tried. I copied the SolrJ Jars contained in the Solr
>> 3.1.0 release
>> to nutch/lib and removed the old ones.
>> So nutch now uses apache-solr-solrj-3.1.0.jar
>> The error however remains the same.
>>
>> Any other suggestions?
>
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Julien Nioche <li...@gmail.com>.
Copying to Nutch lib does not necessarily mean that this is what is being
used until you generate a new job file with 'ant job'. Nutch 1.3 clarifies
this by separating a deployed environment from a local one and it would be a
good idea to use it (it will be released pretty soon anyway).
Am starting to believe that we should upgrade the version of SOLR in 1.3
before it is releases as I expect that many people will be using it.
On 19 April 2011 14:19, Max Stricker <ma...@maxstricker.it> wrote:
>
> > You must upgrade Nutch' two SolrJ jar's to 3.1.
>
> That's what I already tried. I copied the SolrJ Jars contained in the Solr
> 3.1.0 release
> to nutch/lib and removed the old ones.
> So nutch now uses apache-solr-solrj-3.1.0.jar
> The error however remains the same.
>
> Any other suggestions?
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Max Stricker <ma...@maxstricker.it>.
> You must upgrade Nutch' two SolrJ jar's to 3.1.
That's what I already tried. I copied the SolrJ Jars contained in the Solr 3.1.0 release
to nutch/lib and removed the old ones.
So nutch now uses apache-solr-solrj-3.1.0.jar
The error however remains the same.
Any other suggestions?
Re: Nutch 1.2 Solr 3.1.0 solrindex Job failed
Posted by Markus Jelsma <ma...@openindex.io>.
You must upgrade Nutch' two SolrJ jar's to 3.1.
On Tuesday 19 April 2011 14:59:28 Max Stricker wrote:
> Hi all,
>
> I am trying to run Nutch and Solr using the tutorial from [1].
> When indexing the content to Solr using command
> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb crawl/linkdb
> crawl/segments/* I see the following error:
> SolrIndexer: starting at 2011-04-19 03:00:52
> java.io.IOException: Job failed!
>
> The log indicates a Runtime Error when unmarshalling an object:
> 2011-04-19 03:01:26,630 INFO plugin.PluginRepository - Nutch Scoring
> (org.apache.nutch.scoring.ScoringFilter) 2011-04-19 03:01:26,630 INFO
> plugin.PluginRepository - Ontology Model Loader
> (org.apache.nutch.ontology.Ontology) 2011-04-19 03:01:26,634 INFO
> indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter 2011-04-19 03:01:26,635
> INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter 2011-04-19
> 03:01:26,673 INFO solr.SolrMappingReader - source: content dest: content
> 2011-04-19 03:01:26,673 INFO solr.SolrMappingReader - source: site dest:
> site 2011-04-19 03:01:26,673 INFO solr.SolrMappingReader - source: title
> dest: title 2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source:
> host dest: host 2011-04-19 03:01:26,674 INFO solr.SolrMappingReader -
> source: segment dest: segment 2011-04-19 03:01:26,674 INFO
> solr.SolrMappingReader - source: boost dest: boost 2011-04-19 03:01:26,674
> INFO solr.SolrMappingReader - source: digest dest: digest 2011-04-19
> 03:01:26,674 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
> 2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: url dest:
> id 2011-04-19 03:01:26,674 INFO solr.SolrMappingReader - source: url
> dest: url 2011-04-19 03:01:28,544 WARN mapred.LocalJobRunner -
> job_local_0001 java.lang.RuntimeException: Invalid version or the data in
> not in 'javabin' format at
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
> at
> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(Bin
> aryResponseParser.java:39) at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHtt
> pSolrServer.java:466) at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHtt
> pSolrServer.java:243) at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(Abstrac
> tUpdateRequest.java:105) at
> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49) at
> org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:75) at
> org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.j
> ava:48) at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> 2011-04-19 03:01:29,472 ERROR solr.SolrIndexer - java.io.IOException: Job
> failed!
>
> Any ideas what could go wrong here? I strictly followed the mentioned
> tutorial. The index should be fine:
> m@ubuntu:~/Desktop/nutch$ bin/nutch readdb crawl/crawldb -stats
> CrawlDb statistics start: crawl/crawldb
> Statistics for CrawlDb: crawl/crawldb
> TOTAL urls: 284
> retry 0: 284
> min score: 0.0
> avg score: 0.030411972
> max score: 1.208
> status 1 (db_unfetched): 17
> status 2 (db_fetched): 227
> status 3 (db_gone): 40
> CrawlDb statistics: done
>
> regards,
> Max
>
> [1] http://wiki.apache.org/nutch/RunningNutchAndSolr
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350