You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mahesh Charegaonkar <ma...@gmail.com> on 2014/08/20 22:57:06 UTC
Speed up searching on index created using JdbcDirectory
Hi Lucene masters,
I was using lucene couple of years back. We have developed application
which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have
writing and reading data from database.
Over the time data has increased tremendously and that why we are facing
performance issue with searching. I am using Lucene core jar 2.4 and JDK 5.
Could you please suggests good ways to increase the performance.
I have limitation to use JDK5 , so suggest the changes which will be
supported by JDK5 with any new API in lucene.
Thanks
Mahesh
Re: Speed up searching on index created using JdbcDirectory
Posted by Pradeep Bhattiprolu <bp...@gmail.com>.
trace the DB operation Lucene is performing for your search operation and pass on the explain plan to the list.
There is little you can do to tune this from a Lucene point of view but you can probably tune the database to perform this operation faster.
This solution would just get you going for now.
Using Lucene file system indexing is the best option. You can encrypt the indexes if you don't want them to be in clear, but that would add its own overhead.
- Pradeep
> On Aug 23, 2014, at 5:01 PM, Mahesh Charegaonkar <ma...@gmail.com> wrote:
>
> Hi,
>
> My company policy not allowed me to place files on websphere server JVM, so
> thats why we are placing index files in database and searching using
> JDBCDirectory.
>
> I am using 2.4 Lucene API jars API with below ways to search the fields
> from index,
>
>
> Directory dir; = new JdbcDirectory (ds,dialect, indexFileName,true);
> Searcher searcher = new IndexSearcher(dir);
> Hits hits = searcher.search(searchQuery);
> for (int i = 0; i < hits.length(); i++) {
> Document doc = hits.doc(i);
> if(doc.get("Name") != null && !"".equals(doc.get("Name"))){
> ....other biz logic
> }
> }
>
> Thanks
> Mahesh
>
>
>> On Sat, Aug 23, 2014 at 2:48 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>> Hi,
>>
>> there is no need to have an index in a relational database. Lucene indexes
>> are commonly stored as files on local disks. Use FSDirectory subclasses to
>> do this!
>>
>> For more details about performance problem, you should maybe give us more
>> details.
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>>> -----Original Message-----
>>> From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
>>> Sent: Saturday, August 23, 2014 11:42 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Re: Speed up searching on index created using JdbcDirectory
>>>
>>> Thanks Uwe for your response.
>>>
>>> Could you please tell me if i have to still need to keep index in
>> database what
>>> will be good option for searching.
>>>
>>> Thanks
>>> Mahesh
>>>
>>>
>>>> On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>>>
>>>> Don't use JDBCDirectory. It does not scale and has very poor
>> performance.
>>>> This is why it was removed in Lucene 3.
>>>>
>>>> Uwe
>>>>
>>>> -----
>>>> Uwe Schindler
>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>> http://www.thetaphi.de
>>>> eMail: uwe@thetaphi.de
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
>>>>> Sent: Saturday, August 23, 2014 11:12 PM
>>>>> To: java-user@lucene.apache.org
>>>>> Subject: Re: Speed up searching on index created using JdbcDirectory
>>>>>
>>>>> HI All,
>>>>>
>>>>> Please help me out to resolve this issue. Your help is really
>>>> appriciated.
>>>>>
>>>>> Thanks
>>>>> Mahesh
>>>>>
>>>>>
>>>>> On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
>>>>> mahesh.charegaonkar@gmail.com> wrote:
>>>>>
>>>>>> Hi Lucene masters,
>>>>>>
>>>>>> I was using lucene couple of years back. We have developed
>>>>>> application which uses lucene's JdbcDirecory feature. Using
>>>>>> JdbcDirecory we have writing and reading data from database.
>>>>>>
>>>>>> Over the time data has increased tremendously and that why we are
>>>>>> facing performance issue with searching. I am using Lucene core
>>>>>> jar
>>>> 2.4 and
>>>>> JDK 5.
>>>>>> Could you please suggests good ways to increase the performance.
>>>>>>
>>>>>> I have limitation to use JDK5 , so suggest the changes which will
>>>>>> be supported by JDK5 with any new API in lucene.
>>>>>>
>>>>>> Thanks
>>>>>> Mahesh
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Speed up searching on index created using JdbcDirectory
Posted by Mahesh Charegaonkar <ma...@gmail.com>.
Hi,
My company policy not allowed me to place files on websphere server JVM, so
thats why we are placing index files in database and searching using
JDBCDirectory.
I am using 2.4 Lucene API jars API with below ways to search the fields
from index,
Directory dir; = new JdbcDirectory (ds,dialect, indexFileName,true);
Searcher searcher = new IndexSearcher(dir);
Hits hits = searcher.search(searchQuery);
for (int i = 0; i < hits.length(); i++) {
Document doc = hits.doc(i);
if(doc.get("Name") != null && !"".equals(doc.get("Name"))){
....other biz logic
}
}
Thanks
Mahesh
On Sat, Aug 23, 2014 at 2:48 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> there is no need to have an index in a relational database. Lucene indexes
> are commonly stored as files on local disks. Use FSDirectory subclasses to
> do this!
>
> For more details about performance problem, you should maybe give us more
> details.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> > Sent: Saturday, August 23, 2014 11:42 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Speed up searching on index created using JdbcDirectory
> >
> > Thanks Uwe for your response.
> >
> > Could you please tell me if i have to still need to keep index in
> database what
> > will be good option for searching.
> >
> > Thanks
> > Mahesh
> >
> >
> > On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > Don't use JDBCDirectory. It does not scale and has very poor
> performance.
> > > This is why it was removed in Lucene 3.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> > > > Sent: Saturday, August 23, 2014 11:12 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Speed up searching on index created using JdbcDirectory
> > > >
> > > > HI All,
> > > >
> > > > Please help me out to resolve this issue. Your help is really
> > > appriciated.
> > > >
> > > > Thanks
> > > > Mahesh
> > > >
> > > >
> > > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> > > > mahesh.charegaonkar@gmail.com> wrote:
> > > >
> > > > > Hi Lucene masters,
> > > > >
> > > > > I was using lucene couple of years back. We have developed
> > > > > application which uses lucene's JdbcDirecory feature. Using
> > > > > JdbcDirecory we have writing and reading data from database.
> > > > >
> > > > > Over the time data has increased tremendously and that why we are
> > > > > facing performance issue with searching. I am using Lucene core
> > > > > jar
> > > 2.4 and
> > > > JDK 5.
> > > > > Could you please suggests good ways to increase the performance.
> > > > >
> > > > > I have limitation to use JDK5 , so suggest the changes which will
> > > > > be supported by JDK5 with any new API in lucene.
> > > > >
> > > > > Thanks
> > > > > Mahesh
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Speed up searching on index created using JdbcDirectory
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
there is no need to have an index in a relational database. Lucene indexes are commonly stored as files on local disks. Use FSDirectory subclasses to do this!
For more details about performance problem, you should maybe give us more details.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> Sent: Saturday, August 23, 2014 11:42 PM
> To: java-user@lucene.apache.org
> Subject: Re: Speed up searching on index created using JdbcDirectory
>
> Thanks Uwe for your response.
>
> Could you please tell me if i have to still need to keep index in database what
> will be good option for searching.
>
> Thanks
> Mahesh
>
>
> On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Don't use JDBCDirectory. It does not scale and has very poor performance.
> > This is why it was removed in Lucene 3.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> > > Sent: Saturday, August 23, 2014 11:12 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Speed up searching on index created using JdbcDirectory
> > >
> > > HI All,
> > >
> > > Please help me out to resolve this issue. Your help is really
> > appriciated.
> > >
> > > Thanks
> > > Mahesh
> > >
> > >
> > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> > > mahesh.charegaonkar@gmail.com> wrote:
> > >
> > > > Hi Lucene masters,
> > > >
> > > > I was using lucene couple of years back. We have developed
> > > > application which uses lucene's JdbcDirecory feature. Using
> > > > JdbcDirecory we have writing and reading data from database.
> > > >
> > > > Over the time data has increased tremendously and that why we are
> > > > facing performance issue with searching. I am using Lucene core
> > > > jar
> > 2.4 and
> > > JDK 5.
> > > > Could you please suggests good ways to increase the performance.
> > > >
> > > > I have limitation to use JDK5 , so suggest the changes which will
> > > > be supported by JDK5 with any new API in lucene.
> > > >
> > > > Thanks
> > > > Mahesh
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Speed up searching on index created using JdbcDirectory
Posted by Mahesh Charegaonkar <ma...@gmail.com>.
Thanks Uwe for your response.
Could you please tell me if i have to still need to keep index in database
what will be good option for searching.
Thanks
Mahesh
On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Don't use JDBCDirectory. It does not scale and has very poor performance.
> This is why it was removed in Lucene 3.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> > Sent: Saturday, August 23, 2014 11:12 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Speed up searching on index created using JdbcDirectory
> >
> > HI All,
> >
> > Please help me out to resolve this issue. Your help is really
> appriciated.
> >
> > Thanks
> > Mahesh
> >
> >
> > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> > mahesh.charegaonkar@gmail.com> wrote:
> >
> > > Hi Lucene masters,
> > >
> > > I was using lucene couple of years back. We have developed application
> > > which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have
> > > writing and reading data from database.
> > >
> > > Over the time data has increased tremendously and that why we are
> > > facing performance issue with searching. I am using Lucene core jar
> 2.4 and
> > JDK 5.
> > > Could you please suggests good ways to increase the performance.
> > >
> > > I have limitation to use JDK5 , so suggest the changes which will be
> > > supported by JDK5 with any new API in lucene.
> > >
> > > Thanks
> > > Mahesh
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Speed up searching on index created using JdbcDirectory
Posted by Uwe Schindler <uw...@thetaphi.de>.
Don't use JDBCDirectory. It does not scale and has very poor performance. This is why it was removed in Lucene 3.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Mahesh Charegaonkar [mailto:mahesh.charegaonkar@gmail.com]
> Sent: Saturday, August 23, 2014 11:12 PM
> To: java-user@lucene.apache.org
> Subject: Re: Speed up searching on index created using JdbcDirectory
>
> HI All,
>
> Please help me out to resolve this issue. Your help is really appriciated.
>
> Thanks
> Mahesh
>
>
> On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> mahesh.charegaonkar@gmail.com> wrote:
>
> > Hi Lucene masters,
> >
> > I was using lucene couple of years back. We have developed application
> > which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have
> > writing and reading data from database.
> >
> > Over the time data has increased tremendously and that why we are
> > facing performance issue with searching. I am using Lucene core jar 2.4 and
> JDK 5.
> > Could you please suggests good ways to increase the performance.
> >
> > I have limitation to use JDK5 , so suggest the changes which will be
> > supported by JDK5 with any new API in lucene.
> >
> > Thanks
> > Mahesh
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Speed up searching on index created using JdbcDirectory
Posted by Mahesh Charegaonkar <ma...@gmail.com>.
HI All,
Please help me out to resolve this issue. Your help is really appriciated.
Thanks
Mahesh
On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
mahesh.charegaonkar@gmail.com> wrote:
> Hi Lucene masters,
>
> I was using lucene couple of years back. We have developed application
> which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have
> writing and reading data from database.
>
> Over the time data has increased tremendously and that why we are facing
> performance issue with searching. I am using Lucene core jar 2.4 and JDK 5.
> Could you please suggests good ways to increase the performance.
>
> I have limitation to use JDK5 , so suggest the changes which will be
> supported by JDK5 with any new API in lucene.
>
> Thanks
> Mahesh
>