You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Radha C." <cr...@ceiindia.com> on 2009/03/04 15:02:59 UTC
How to search the database tables using solr.
Hi,
I am working in a software concern. We are having some R&D base work like
making use of solr search tool for searching our Oracle database
content. I read through the documnets provided in the site, there I found
the solr supports only the document search and we have to provide the
document in the xml with the whole content that we need to search. But Is
there any way to index our database content and search the indexed database
content.
Any help will be appriciated. Thanks.
Re: How to search the database tables using solr.
Posted by Glen Newton <gl...@gmail.com>.
Also take a look at LuSql:
http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/3/4 Shalin Shekhar Mangar <sh...@gmail.com>:
> On Wed, Mar 4, 2009 at 7:32 PM, Radha C. <cr...@ceiindia.com> wrote:
>
>> Hi,
>>
>> I am working in a software concern. We are having some R&D base work like
>> making use of solr search tool for searching our Oracle database
>> content. I read through the documnets provided in the site, there I found
>> the solr supports only the document search and we have to provide the
>> document in the xml with the whole content that we need to search. But Is
>> there any way to index our database content and search the indexed database
>> content.
>>
>
> Take a look at DataImportHandler.
>
> http://wiki.apache.org/solr/DataImportHandler
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
-
Re: How to search the database tables using solr.
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Mar 5, 2009 at 10:09 PM, Radha C. <cr...@ceiindia.com> wrote:
> I want to understand fully what is configured and how to configure , So I
> tried to index my local MySql DB directly with one simple table called
> persons in it.
> But I am getting lot of errors. the following are the steps I did,
> 1. downloaded solr-2009-03-03.zip
> <http://people.apache.org/builds/lucene/solr/nightly/solr-2009-03-03.zip>
> distribution and extracted to d:/solrtemp/
> 2. copied the example/solr to d:/solr ( this is my solr home ) for my
> application template.
> 3. set this solr home insidecatalina-home/localconfig/solr.xml
> 4. create the data-config under $solrhome/config
> here is my dataconfig file
> <dataConfig>
> <dataSource driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/test" user="xxx" password="xxxx" />
> <document name="employees">
> <entity name="persons" pk="personid" query="select * from
> persons" deltaQuery="select personID from persons where last_modified >
> '${dataimporter.last_index_time}'">
> <field column="firstname"/>
> <field column="lastName"/>
> <field column="age"/>
> </entity>
> </document>
> </dataConfig>
>
> I am not sure what needs to be done next. I am getting error when I start
> tomcat.
Just like you write DDL for databases, you need to define a schema in Solr
as well. The missing step is to change solr's schema.xml according to the
fields you want to index. Look at http://wiki.apache.org/solr/SchemaXml for
details.
The errors are a way of warning you that there are certain required fields
in schema which are never filled by the configuration in the data-config.
>
>
> Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImportHandler
> processConfiguration
> INFO: Processing configuration from solrconfig.xml:
> {config=data-config.xml}
> Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: id is a required field in SolrSchema . But not found in DataConfig
> Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImportHandler
> inform
> SEVERE: Exception while loading DataImporter
> org.apache.solr.handler.dataimport.DataImportHandlerException: There are
> errors in the Schema
> The field :age present in DataConfig does not have a counterpart in Solr
> Schema
> The field :firstname present in DataConfig does not have a counterpart in
> Solr Schema
> The field :personid present in DataConfig does not have a counterpart in
> Solr Schema
> The field :lastName present in DataConfig does not have a counterpart in
> Solr Schema
> at
>
> org.apache.solr.handler.dataimport.DataImporter.<init>(DataImporter.java:108
> )
> at
>
> org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
> r.java:95)
> at
> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388)
> at
>
> org.apache.solr.core.SolrCore.<init>(SolrCore.java:571).....................
> .
> Mar 5, 2009 9:10:28 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
> org.apache.solr.common.SolrException: FATAL: Could not create importer.
> DataImporter config invalid
> at
>
> org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
> r.java:103)
> at
> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388)
>
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> There are errors in the Schema
> The field :age present in DataConfig does not have a counterpart in Solr
> Schema
> The field :firstname present in DataConfig does not have a counterpart in
> Solr Schema
> The field :personid present in DataConfig does not have a counterpart in
> Solr Schema
> The field :lastName present in DataConfig does not have a counterpart in
> Solr Schema
> at
>
> org.apache.solr.handler.dataimport.DataImporter.<init>(DataImporter.java:108
> )
> at
>
> org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
> r.java:95)
> ... 31 more
> Mar 5, 2009 9:10:28 PM org.apache.solr.core.QuerySenderListener newSearcher
>
> I want to write a small seperate application without relying on examples,
> can you provide me some useful steps how to do that.
>
> Thanks
>
> _____
>
> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
> Sent: Thursday, March 05, 2009 5:16 PM
> To: solr-user@lucene.apache.org; cradha@ceiindia.com
> Subject: Re: How to search the database tables using solr.
>
>
> On Thu, Mar 5, 2009 at 4:42 PM, Radha C. <cr...@ceiindia.com> wrote:
>
>
>
> Hi,
>
> I am newbie for solr search engin. I don't find any juicy information on
> how
> to configure the ORACLE data base to index the tables using solr search
> engin. There are huge documents spread over wiki pages. I need some core
> information.
> I am using Apache-tomcat 5.5.26, and oracle 9i. Can you please provide me a
> brief details for installing and configuring the database with solr and how
> to indexing the rows. Your help will really save lot of my time.
>
>
>
>
> Have you been able to run the example/example-DIH demo? If yes, using
> Oracle
> database instead of the HSQLDB used in the example is easy. Substitute the
> driver name, username, password with the appropriate values for your
> database. Make sure you add the oracle driver's jar file to $solr_home/lib.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
Regards,
Shalin Shekhar Mangar.
RE: How to search the database tables using solr.
Posted by "Radha C." <cr...@ceiindia.com>.
Shalin,
I did not run the examples because in the demo everything is already
configured and built in, so the demo will run properly.
So I am not clear about how the example is working and what is the
configuration done for that, Where to start if I write a small new database
search.
I want to understand fully what is configured and how to configure , So I
tried to index my local MySql DB directly with one simple table called
persons in it.
But I am getting lot of errors. the following are the steps I did,
1. downloaded solr-2009-03-03.zip
<http://people.apache.org/builds/lucene/solr/nightly/solr-2009-03-03.zip>
distribution and extracted to d:/solrtemp/
2. copied the example/solr to d:/solr ( this is my solr home ) for my
application template.
3. set this solr home insidecatalina-home/localconfig/solr.xml
4. create the data-config under $solrhome/config
here is my dataconfig file
<dataConfig>
<dataSource driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/test" user="xxx" password="xxxx" />
<document name="employees">
<entity name="persons" pk="personid" query="select * from
persons" deltaQuery="select personID from persons where last_modified >
'${dataimporter.last_index_time}'">
<field column="firstname"/>
<field column="lastName"/>
<field column="age"/>
</entity>
</document>
</dataConfig>
I am not sure what needs to be done next. I am getting error when I start
tomcat.
Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImportHandler
processConfiguration
INFO: Processing configuration from solrconfig.xml: {config=data-config.xml}
Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
INFO: Data Configuration loaded successfully
Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
Mar 5, 2009 9:10:28 PM org.apache.solr.handler.dataimport.DataImportHandler
inform
SEVERE: Exception while loading DataImporter
org.apache.solr.handler.dataimport.DataImportHandlerException: There are
errors in the Schema
The field :age present in DataConfig does not have a counterpart in Solr
Schema
The field :firstname present in DataConfig does not have a counterpart in
Solr Schema
The field :personid present in DataConfig does not have a counterpart in
Solr Schema
The field :lastName present in DataConfig does not have a counterpart in
Solr Schema
at
org.apache.solr.handler.dataimport.DataImporter.<init>(DataImporter.java:108
)
at
org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
r.java:95)
at
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388)
at
org.apache.solr.core.SolrCore.<init>(SolrCore.java:571).....................
.
Mar 5, 2009 9:10:28 PM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property
org.apache.solr.common.SolrException: FATAL: Could not create importer.
DataImporter config invalid
at
org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
r.java:103)
at
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
There are errors in the Schema
The field :age present in DataConfig does not have a counterpart in Solr
Schema
The field :firstname present in DataConfig does not have a counterpart in
Solr Schema
The field :personid present in DataConfig does not have a counterpart in
Solr Schema
The field :lastName present in DataConfig does not have a counterpart in
Solr Schema
at
org.apache.solr.handler.dataimport.DataImporter.<init>(DataImporter.java:108
)
at
org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandle
r.java:95)
... 31 more
Mar 5, 2009 9:10:28 PM org.apache.solr.core.QuerySenderListener newSearcher
I want to write a small seperate application without relying on examples,
can you provide me some useful steps how to do that.
Thanks
_____
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
Sent: Thursday, March 05, 2009 5:16 PM
To: solr-user@lucene.apache.org; cradha@ceiindia.com
Subject: Re: How to search the database tables using solr.
On Thu, Mar 5, 2009 at 4:42 PM, Radha C. <cr...@ceiindia.com> wrote:
Hi,
I am newbie for solr search engin. I don't find any juicy information on how
to configure the ORACLE data base to index the tables using solr search
engin. There are huge documents spread over wiki pages. I need some core
information.
I am using Apache-tomcat 5.5.26, and oracle 9i. Can you please provide me a
brief details for installing and configuring the database with solr and how
to indexing the rows. Your help will really save lot of my time.
Have you been able to run the example/example-DIH demo? If yes, using Oracle
database instead of the HSQLDB used in the example is easy. Substitute the
driver name, username, password with the appropriate values for your
database. Make sure you add the oracle driver's jar file to $solr_home/lib.
--
Regards,
Shalin Shekhar Mangar.
Re: How to search the database tables using solr.
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Mar 5, 2009 at 4:42 PM, Radha C. <cr...@ceiindia.com> wrote:
>
> Hi,
>
> I am newbie for solr search engin. I don't find any juicy information on
> how
> to configure the ORACLE data base to index the tables using solr search
> engin. There are huge documents spread over wiki pages. I need some core
> information.
> I am using Apache-tomcat 5.5.26, and oracle 9i. Can you please provide me a
> brief details for installing and configuring the database with solr and how
> to indexing the rows. Your help will really save lot of my time.
>
>
Have you been able to run the example/example-DIH demo? If yes, using Oracle
database instead of the HSQLDB used in the example is easy. Substitute the
driver name, username, password with the appropriate values for your
database. Make sure you add the oracle driver's jar file to $solr_home/lib.
--
Regards,
Shalin Shekhar Mangar.
Re: How to search the database tables using solr.
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
it depends on how fast your DB can give out data through jdbc . the
best thing is to just run and see.
--Noble
On Thu, Mar 5, 2009 at 1:13 PM, Venu Mittal <me...@yahoo.com> wrote:
> Does anybody has any stats to share on how much time does DataImportHandler takes to index a given set of data ?
>
> I am currently indexing 18 millions rows in 1.5 - 2 hours by sending xmls to solr.
>
>
>
> ________________________________
> From: Shalin Shekhar Mangar <sh...@gmail.com>
> To: solr-user@lucene.apache.org; cradha@ceiindia.com
> Sent: Wednesday, March 4, 2009 8:15:07 AM
> Subject: Re: How to search the database tables using solr.
>
> On Wed, Mar 4, 2009 at 7:51 PM, Radha C. <cr...@ceiindia.com> wrote:
>
>> Thanks Shalin,
>>
>> We just stepped on solr. This information is very much useful for me. But
>> before that I want some clear details about where to start..
>> I want to test this in my local environment, so I need some basic
>> information about how to start using this ( database and solr ). Do you
>> have
>> some information on this?
>>
>
> I think the easiest way is to start using Solr is with the embedded jetty
> container. Modify the example/conf/schema.xml file and add your own fields
> etc. Read through the DataImportHandler wiki page and at the
> example/example-DIH directory in the solr zip/tarball.
>
> If you have a specific doubt/question, ask on the list.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
>
>
--
--Noble Paul
RE: How to search the database tables using solr.
Posted by "Radha C." <cr...@ceiindia.com>.
Hi,
I am newbie for solr search engin. I don't find any juicy information on how
to configure the ORACLE data base to index the tables using solr search
engin. There are huge documents spread over wiki pages. I need some core
information.
I am using Apache-tomcat 5.5.26, and oracle 9i. Can you please provide me a
brief details for installing and configuring the database with solr and how
to indexing the rows. Your help will really save lot of my time.
-----Original Message-----
From: Venu Mittal [mailto:metale666@yahoo.com]
Sent: Thursday, March 05, 2009 1:14 PM
To: solr-user@lucene.apache.org
Subject: Re: How to search the database tables using solr.
Does anybody has any stats to share on how much time does DataImportHandler
takes to index a given set of data ?
I am currently indexing 18 millions rows in 1.5 - 2 hours by sending xmls to
solr.
________________________________
From: Shalin Shekhar Mangar <sh...@gmail.com>
To: solr-user@lucene.apache.org; cradha@ceiindia.com
Sent: Wednesday, March 4, 2009 8:15:07 AM
Subject: Re: How to search the database tables using solr.
On Wed, Mar 4, 2009 at 7:51 PM, Radha C. <cr...@ceiindia.com> wrote:
> Thanks Shalin,
>
> We just stepped on solr. This information is very much useful for me.
> But before that I want some clear details about where to start..
> I want to test this in my local environment, so I need some basic
> information about how to start using this ( database and solr ). Do
> you have some information on this?
>
I think the easiest way is to start using Solr is with the embedded jetty
container. Modify the example/conf/schema.xml file and add your own fields
etc. Read through the DataImportHandler wiki page and at the
example/example-DIH directory in the solr zip/tarball.
If you have a specific doubt/question, ask on the list.
--
Regards,
Shalin Shekhar Mangar.
Re: How to search the database tables using solr.
Posted by Venu Mittal <me...@yahoo.com>.
Does anybody has any stats to share on how much time does DataImportHandler takes to index a given set of data ?
I am currently indexing 18 millions rows in 1.5 - 2 hours by sending xmls to solr.
________________________________
From: Shalin Shekhar Mangar <sh...@gmail.com>
To: solr-user@lucene.apache.org; cradha@ceiindia.com
Sent: Wednesday, March 4, 2009 8:15:07 AM
Subject: Re: How to search the database tables using solr.
On Wed, Mar 4, 2009 at 7:51 PM, Radha C. <cr...@ceiindia.com> wrote:
> Thanks Shalin,
>
> We just stepped on solr. This information is very much useful for me. But
> before that I want some clear details about where to start..
> I want to test this in my local environment, so I need some basic
> information about how to start using this ( database and solr ). Do you
> have
> some information on this?
>
I think the easiest way is to start using Solr is with the embedded jetty
container. Modify the example/conf/schema.xml file and add your own fields
etc. Read through the DataImportHandler wiki page and at the
example/example-DIH directory in the solr zip/tarball.
If you have a specific doubt/question, ask on the list.
--
Regards,
Shalin Shekhar Mangar.
Re: How to search the database tables using solr.
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Mar 4, 2009 at 7:51 PM, Radha C. <cr...@ceiindia.com> wrote:
> Thanks Shalin,
>
> We just stepped on solr. This information is very much useful for me. But
> before that I want some clear details about where to start..
> I want to test this in my local environment, so I need some basic
> information about how to start using this ( database and solr ). Do you
> have
> some information on this?
>
I think the easiest way is to start using Solr is with the embedded jetty
container. Modify the example/conf/schema.xml file and add your own fields
etc. Read through the DataImportHandler wiki page and at the
example/example-DIH directory in the solr zip/tarball.
If you have a specific doubt/question, ask on the list.
--
Regards,
Shalin Shekhar Mangar.
RE: How to search the database tables using solr.
Posted by "Radha C." <cr...@ceiindia.com>.
Thanks Shalin,
We just stepped on solr. This information is very much useful for me. But
before that I want some clear details about where to start..
I want to test this in my local environment, so I need some basic
information about how to start using this ( database and solr ). Do you have
some information on this?
Thanks in advance.
_____
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
Sent: Wednesday, March 04, 2009 7:48 PM
To: solr-user@lucene.apache.org; cradha@ceiindia.com
Subject: Re: How to search the database tables using solr.
On Wed, Mar 4, 2009 at 7:32 PM, Radha C. <cr...@ceiindia.com> wrote:
Hi,
I am working in a software concern. We are having some R&D base work like
making use of solr search tool for searching our Oracle database
content. I read through the documnets provided in the site, there I found
the solr supports only the document search and we have to provide the
document in the xml with the whole content that we need to search. But Is
there any way to index our database content and search the indexed database
content.
Take a look at DataImportHandler.
http://wiki.apache.org/solr/DataImportHandler
--
Regards,
Shalin Shekhar Mangar.
Re: How to search the database tables using solr.
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Mar 4, 2009 at 7:32 PM, Radha C. <cr...@ceiindia.com> wrote:
> Hi,
>
> I am working in a software concern. We are having some R&D base work like
> making use of solr search tool for searching our Oracle database
> content. I read through the documnets provided in the site, there I found
> the solr supports only the document search and we have to provide the
> document in the xml with the whole content that we need to search. But Is
> there any way to index our database content and search the indexed database
> content.
>
Take a look at DataImportHandler.
http://wiki.apache.org/solr/DataImportHandler
--
Regards,
Shalin Shekhar Mangar.