You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by al...@aim.com on 2009/05/11 21:14:07 UTC
how to manually add data to indexes generated by nutch-1.0 using solr
Hello,
I had? Nutch -1.0 to crawl fetch and index a lot of files. Then I needed to?
index a few files also. But I know keywords for those files and their?
locations. I need to add them manually. I took a look to two tutorials on the wiki, but did not find any info about this issue.
Is there a tutorial on, step by step procedure of adding data to? nutch index using solr? manually?
Thanks in advance.
Alex.
Re: Adding data from nutch to a Solr index
Posted by Andrzej Bialecki <ab...@getopt.org>.
Sönke Goldbeck wrote:
> Alright, first post to this list and I hope the question
> is not too stupid or misplaced ...
>
> what I currently have:
> - a nicely working Solr 1.3 index with information about some
> entities e.g. organisations, indexed from an RDBMS. Many of these
> entities have an URL pointing at further information, e.g. the
> website of an institute or company.
>
> - an installation of nutch 0.9 with which I can crawl for the
> URLs that I can extract from the RDBMS mentioned above and put
> into a seed file
>
> - tutorials about how to put crawled and indexed data from
> nutch 1.0 (which I could install w/o problems) into a separate
> Solr index
>
>
> what I want:
> - combine the indexed information from the RDBMS and the website
> in one Solr index so that I can search both in one and with the
> capability of using all the Solr features. E.g. having the following
> (example) fields in one document:
>
> <doc>
> <name-from-RDBMS>
> <indexed-content-from-RDBMS>
> <indexed-content-from-website>
> <URL>
> <...>
> </doc>
I believe that this kind of document merging is not possible (at least
not easily) - you have to assemble the whole document before you index
it in Solr.
If these documents use the same primary key (I guess they do, otherwise
how would you merge them...) then you can do the merging in your
front-end application, which would have to submit the main query to
Solr, and then for each Solr document on the list of results it would
retrieve a Nutch document (using NutchBean API).
(The not so easy way involves writing a SearchComponent that does the
latter part of that process on the Solr side.)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Adding data from nutch to a Solr index
Posted by Sönke Goldbeck <go...@avail.de>.
Alright, first post to this list and I hope the question
is not too stupid or misplaced ...
what I currently have:
- a nicely working Solr 1.3 index with information about some
entities e.g. organisations, indexed from an RDBMS. Many of these
entities have an URL pointing at further information, e.g. the
website of an institute or company.
- an installation of nutch 0.9 with which I can crawl for the
URLs that I can extract from the RDBMS mentioned above and put
into a seed file
- tutorials about how to put crawled and indexed data from
nutch 1.0 (which I could install w/o problems) into a separate
Solr index
what I want:
- combine the indexed information from the RDBMS and the website
in one Solr index so that I can search both in one and with the
capability of using all the Solr features. E.g. having the following
(example) fields in one document:
<doc>
<name-from-RDBMS>
<indexed-content-from-RDBMS>
<indexed-content-from-website>
<URL>
<...>
</doc>
Any input appreciated!
Cheers, Sönke
Re: how to manually add data to indexes generated by nutch-1.0 using solr
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Try a search for *:* and see if you get results for that. If so, you
have your documents indexed, but you need to dig into things like
query parser configuration and analysis to see why things aren't
matching. Perhaps you're not querying the field you think you are?
Erik
On May 13, 2009, at 1:15 PM, alxsss@aim.com wrote:
>
> I forget to say that when I do
>
> curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --
> data-binary '<commit waitFlush="false" waitSearcher="false"/>'
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="responseHeader"><int name="status">0</int><int
> name="QTime">453</int></lst>
> </response>
>
>
> and search for added keywords gives 0 results. Does status 0 mean
> that addition was successful?
>
> Thanks.
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: Erik Hatcher <er...@ehatchersolutions.com>
> To: solr-user@lucene.apache.org
> Sent: Tue, 12 May 2009 6:48 pm
> Subject: Re: how to manually add data to indexes generated by
> nutch-1.0 using solr
>
>
>
>
>
>
>
>
>
> send a <commit/> request afterwards, or you can add ?commit=true to
> the /update request with the adds.?
> ?
>
> ? Erik?
> ?
>
> On May 12, 2009, at 8:57 PM, alxsss@aim.com wrote:?
> ?
>
>> ?
>
>> Tried to add a new record using?
>
>> ?
>
>> ?
>
>> ?
>
>> curl http://localhost:8983/solr/update -H "Content-Type: text/xml"
>> --> data-binary '<add>?
>
>> <doc boost="2.5">?
>
>> <field name="segment">20090512170318</field>?
>
>> <field name="digest">86937aaee8e748ac3007ed8b66477624</field>?
>
>> <field name="boost">0.21189615</field>?
>
>> <field name="url">test.com</field>?
>
>> <field name="title">test test</field>?
>
>> <field name="tstamp"> 20090513003210909</field>?
>
>> </doc> </add>'?
>
>> ?
>
>> I get?
>
>> ?
>
>> <?xml version="1.0" encoding="UTF-8"?>?
>
>> <response>?
>
>> <lst name="responseHeader"><int name="status">0</int><int >
>> name="QTime">71</int></lst>?
>
>> </response>?
>
>> ?
>
>> ?
>
>> and added records are not found in the search.?
>
>> ?
>
>> Any ideas what went wrong??
>
>> ?
>
>> ?
>
>> Thanks.?
>
>> Alex.?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> -----Original Message-----?
>
>> From: alxsss@aim.com?
>
>> To: solr-user@lucene.apache.org?
>
>> Sent: Mon, 11 May 2009 12:14 pm?
>
>> Subject: how to manually add data to indexes generated by nutch-1.0
>> > using solr?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> Hello,?
>
>> ?
>
>> I had? Nutch -1.0 to crawl fetch and index a lot of files. Then I >
>> needed to??
>
>> ?
>
>> index a few files also. But I know keywords for those files and
>> their??
>
>> locations. I need to add them manually. I took a look to two >
>> tutorials on the?
>
>> wiki, but did not find any info about this issue.?
>
>> Is there a tutorial on, step by step procedure of adding data to? >
>> nutch index?
>
>> using solr? manually??
>
>> ?
>
>> Thanks in advance.?
>
>> Alex.?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
> ?
>
>
>
>
>
Re: how to manually add data to indexes generated by nutch-1.0 using solr
Posted by al...@aim.com.
I forget to say that when I do
curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --data-binary '<commit waitFlush="false" waitSearcher="false"/>'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">453</int></lst>
</response>
and search for added keywords gives 0 results. Does status 0 mean that addition was successful?
Thanks.
Alex.
-----Original Message-----
From: Erik Hatcher <er...@ehatchersolutions.com>
To: solr-user@lucene.apache.org
Sent: Tue, 12 May 2009 6:48 pm
Subject: Re: how to manually add data to indexes generated by nutch-1.0 using solr
send a <commit/> request afterwards, or you can add ?commit=true to the /update request with the adds.?
?
? Erik?
?
On May 12, 2009, at 8:57 PM, alxsss@aim.com wrote:?
?
>?
> Tried to add a new record using?
>?
>?
>?
> curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --> data-binary '<add>?
> <doc boost="2.5">?
> <field name="segment">20090512170318</field>?
> <field name="digest">86937aaee8e748ac3007ed8b66477624</field>?
> <field name="boost">0.21189615</field>?
> <field name="url">test.com</field>?
> <field name="title">test test</field>?
> <field name="tstamp"> 20090513003210909</field>?
> </doc> </add>'?
>?
> I get?
>?
> <?xml version="1.0" encoding="UTF-8"?>?
> <response>?
> <lst name="responseHeader"><int name="status">0</int><int > name="QTime">71</int></lst>?
> </response>?
>?
>?
> and added records are not found in the search.?
>?
> Any ideas what went wrong??
>?
>?
> Thanks.?
> Alex.?
>?
>?
>?
>?
> -----Original Message-----?
> From: alxsss@aim.com?
> To: solr-user@lucene.apache.org?
> Sent: Mon, 11 May 2009 12:14 pm?
> Subject: how to manually add data to indexes generated by nutch-1.0 > using solr?
>?
>?
>?
>?
>?
>?
>?
>?
>?
>?
> Hello,?
>?
> I had? Nutch -1.0 to crawl fetch and index a lot of files. Then I > needed to??
>?
> index a few files also. But I know keywords for those files and their??
> locations. I need to add them manually. I took a look to two > tutorials on the?
> wiki, but did not find any info about this issue.?
> Is there a tutorial on, step by step procedure of adding data to? > nutch index?
> using solr? manually??
>?
> Thanks in advance.?
> Alex.?
>?
>?
>?
>?
>?
?
Re: how to manually add data to indexes generated by nutch-1.0 using solr
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
send a <commit/> request afterwards, or you can add ?commit=true to
the /update request with the adds.
Erik
On May 12, 2009, at 8:57 PM, alxsss@aim.com wrote:
>
> Tried to add a new record using
>
>
>
> curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --
> data-binary '<add>
> <doc boost="2.5">
> <field name="segment">20090512170318</field>
> <field name="digest">86937aaee8e748ac3007ed8b66477624</field>
> <field name="boost">0.21189615</field>
> <field name="url">test.com</field>
> <field name="title">test test</field>
> <field name="tstamp"> 20090513003210909</field>
> </doc> </add>'
>
> I get
>
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="responseHeader"><int name="status">0</int><int
> name="QTime">71</int></lst>
> </response>
>
>
> and added records are not found in the search.
>
> Any ideas what went wrong?
>
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: alxsss@aim.com
> To: solr-user@lucene.apache.org
> Sent: Mon, 11 May 2009 12:14 pm
> Subject: how to manually add data to indexes generated by nutch-1.0
> using solr
>
>
>
>
>
>
>
>
>
>
> Hello,
>
> I had? Nutch -1.0 to crawl fetch and index a lot of files. Then I
> needed to?
>
> index a few files also. But I know keywords for those files and their?
> locations. I need to add them manually. I took a look to two
> tutorials on the
> wiki, but did not find any info about this issue.
> Is there a tutorial on, step by step procedure of adding data to?
> nutch index
> using solr? manually?
>
> Thanks in advance.
> Alex.
>
>
>
>
>
Re: how to manually add data to indexes generated by nutch-1.0 using solr
Posted by al...@aim.com.
Tried to add a new record using
curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --data-binary '<add>
<doc boost="2.5">
<field name="segment">20090512170318</field>
<field name="digest">86937aaee8e748ac3007ed8b66477624</field>
<field name="boost">0.21189615</field>
<field name="url">test.com</field>
<field name="title">test test</field>
<field name="tstamp"> 20090513003210909</field>
</doc> </add>'
I get
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">71</int></lst>
</response>
and added records are not found in the search.
Any ideas what went wrong?
Thanks.
Alex.
-----Original Message-----
From: alxsss@aim.com
To: solr-user@lucene.apache.org
Sent: Mon, 11 May 2009 12:14 pm
Subject: how to manually add data to indexes generated by nutch-1.0 using solr
Hello,
I had? Nutch -1.0 to crawl fetch and index a lot of files. Then I needed to?
index a few files also. But I know keywords for those files and their?
locations. I need to add them manually. I took a look to two tutorials on the
wiki, but did not find any info about this issue.
Is there a tutorial on, step by step procedure of adding data to? nutch index
using solr? manually?
Thanks in advance.
Alex.