You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Denis Sinner <de...@dkd.de> on 2012/01/30 10:29:35 UTC
Index difference between crawl and solrindex command
Hello,
I have Nutch setup on a Linux server with Solr running on the same server. When i try to crawl some websites i get a job fail at the indexing part of the crawl:
2012-01-30 10:11:29,445 INFO solr.SolrWriter - Adding 1 documents
2012-01-30 10:11:29,698 WARN mapred.LocalJobRunner - job_local_0009
org.apache.solr.common.SolrException: Not Found
Not Found
request: http://127.0.0.1:8080/solr_3-5/searchdkdde_en/update?wt=javabin&version=2
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:93)
at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
2012-01-30 10:11:30,362 ERROR solr.SolrIndexer - java.io.IOException: Job failed!
Now, when i call the 'solrindex' command with the freshly created crawl Folders as arguments, the document gets indexed.
So - what is the difference between the 'solrindex' command and the indexing part of the 'crawl' command? What could be the reason why the latter doesn't work?
My solution would be to write a shell script that simulates the 'crawl' command by calling the single inject/fetch/index/... commands.
--
[Entwickler]
dkd Internet Service GmbH
development // kommunikation // design
Kaiserstraße 73
60329 Frankfurt/Main
fon: +49 69 2475218-0
fax: +49 69 2475218-99
e-mail: denis.sinner@dkd.de
twitter: http://twitter.com/dkd_de
facebook: http://www.facebook.com/www.dkd.de
web: http://www.dkd.de
Registergericht: Amtsgericht Frankfurt am Main
Registernummer: HRB 45590
Geschäftsführer: Olivier Dobberkau, Søren Schaffstein, Götz Wegenast, Christian Zabanski
Aktuelle Projekte:
http://www.spielwarenmesse-eg.de – Relaunch & Responsive Design (TYPO3)
http://www.horsch.com – Relaunch Website (TYPO3)
http://www.dosb.de – Refresh Website (TYPO3)
Re: Index difference between crawl and solrindex command
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 30/01/2012 12:38, Denis Sinner wrote:
> crawl:
>
> bin/nutch crawl urls -solr http://127.0.0.1:8080/solr_3-5/searchDkdDe_en -depth 1
>
> fails at indexing, then
>
> bin/nutch solrindex http://127.0.0.1:8080/solr_3-5/searchDkdDe_en crawl-20120130123251/crawldb/ -linkdb crawl-20120130123251/linkdb/ crawl-20120130123251/segments/*
>
> works!
>
> btw. in the logs, when crawl requests the solr server, the url is lowercased, where in solrindex it is not.
Ah, indeed - looks like a bug in the Crawl command. See line 87 of
Crawl.java - just replace it with a simple assignment, like this:
solrUrl = args[i + 1];
And rebuild. I'll file a JIRA issue and will fix it in Nutch trunk. Thanks!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Index difference between crawl and solrindex command
Posted by Denis Sinner <de...@dkd.de>.
crawl:
bin/nutch crawl urls -solr http://127.0.0.1:8080/solr_3-5/searchDkdDe_en -depth 1
fails at indexing, then
bin/nutch solrindex http://127.0.0.1:8080/solr_3-5/searchDkdDe_en crawl-20120130123251/crawldb/ -linkdb crawl-20120130123251/linkdb/ crawl-20120130123251/segments/*
works!
btw. in the logs, when crawl requests the solr server, the url is lowercased, where in solrindex it is not.
--
[Entwickler]
dkd Internet Service GmbH
development // kommunikation // design
Kaiserstraße 73
60329 Frankfurt/Main
fon: +49 69 2475218-0
fax: +49 69 2475218-99
e-mail: denis.sinner@dkd.de
twitter: http://twitter.com/dkd_de
facebook: http://www.facebook.com/www.dkd.de
web: http://www.dkd.de
Registergericht: Amtsgericht Frankfurt am Main
Registernummer: HRB 45590
Geschäftsführer: Olivier Dobberkau, Søren Schaffstein, Götz Wegenast, Christian Zabanski
Aktuelle Projekte:
http://www.spielwarenmesse-eg.de – Relaunch & Responsive Design (TYPO3)
http://www.horsch.com – Relaunch Website (TYPO3)
http://www.dosb.de – Refresh Website (TYPO3)
Am 30.01.2012 um 11:08 schrieb Andrzej Bialecki:
>
Re: Index difference between crawl and solrindex command
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 30/01/2012 11:19, Denis Sinner wrote:
> Thanks,
>
>> Most likely it uses a different Solr url (perhaps without the .../update part?).
>
> Doesn't seem so, if i call the solrindex command with a wrong url, i get a similar error with the following call:
>
> request: http://127.0.0.1:8080/solr_3-5/searchDkdDe_/update?wt=javabin&version=2
Can you copy&paste the exact commands that you used in each case?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Index difference between crawl and solrindex command
Posted by Denis Sinner <de...@dkd.de>.
Thanks,
> Most likely it uses a different Solr url (perhaps without the .../update part?).
Doesn't seem so, if i call the solrindex command with a wrong url, i get a similar error with the following call:
request: http://127.0.0.1:8080/solr_3-5/searchDkdDe_/update?wt=javabin&version=2
--
[Entwickler]
dkd Internet Service GmbH
development // kommunikation // design
Kaiserstraße 73
60329 Frankfurt/Main
fon: +49 69 2475218-0
fax: +49 69 2475218-99
e-mail: denis.sinner@dkd.de
twitter: http://twitter.com/dkd_de
facebook: http://www.facebook.com/www.dkd.de
web: http://www.dkd.de
Registergericht: Amtsgericht Frankfurt am Main
Registernummer: HRB 45590
Geschäftsführer: Olivier Dobberkau, Søren Schaffstein, Götz Wegenast, Christian Zabanski
Aktuelle Projekte:
http://www.spielwarenmesse-eg.de – Relaunch & Responsive Design (TYPO3)
http://www.horsch.com – Relaunch Website (TYPO3)
http://www.dosb.de – Refresh Website (TYPO3)
Re: Index difference between crawl and solrindex command
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 30/01/2012 10:29, Denis Sinner wrote:
> Hello,
> I have Nutch setup on a Linux server with Solr running on the same server. When i try to crawl some websites i get a job fail at the indexing part of the crawl:
>
> 2012-01-30 10:11:29,445 INFO solr.SolrWriter - Adding 1 documents
> 2012-01-30 10:11:29,698 WARN mapred.LocalJobRunner - job_local_0009
> org.apache.solr.common.SolrException: Not Found
>
> Not Found
>
> request: http://127.0.0.1:8080/solr_3-5/searchdkdde_en/update?wt=javabin&version=2
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
> at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
> at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:49)
> at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:93)
> at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
> 2012-01-30 10:11:30,362 ERROR solr.SolrIndexer - java.io.IOException: Job failed!
This message means that either:
* there is no solr instance listening on that URL, or
* there is no such handler using that URL, i.e. the path
/solr_3-5/searchdkdde_en/update in fact is not a Solr update handler.
> Now, when i call the 'solrindex' command with the freshly created crawl Folders as arguments, the document gets indexed.
Most likely it uses a different Solr url (perhaps without the .../update
part?).
>
> So - what is the difference between the 'solrindex' command and the indexing part of the 'crawl' command? What could be the reason why the latter doesn't work?
> My solution would be to write a shell script that simulates the 'crawl' command by calling the single inject/fetch/index/... commands.
>
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com