You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2009/11/16 23:17:24 UTC
Re: Solr - Load Increasing.
Hi,
Your autoCommit settings are very aggressive. I'm guessing that's what's causing the CPU load.
btw. what is "laks"?
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: kalidoss <ka...@sifycorp.com>
> To: solr-user@lucene.apache.org
> Sent: Mon, November 16, 2009 9:11:21 AM
> Subject: Solr - Load Increasing.
>
> Hi All.
>
> My server solr box cpu utilization increasing b/w 60 to 90% and some time
> solr is getting down and we are restarting it manually.
>
> No of documents in solr 30 laks.
> No of add/update requrest solr 30 thousand / day. Avg of every 30 minutes
> around 500 writes.
> No of search request 9laks / day.
> Size of the data directory: 4gb.
>
>
> My system ram is 8gb.
> System available space 12gb.
> processor Family: Pentium Pro
>
> Our solr data size can be increase in number like 90 laks. and writes per day
> will be around 1laks. - Hope its possible by solr.
>
> For write commit i have configured like
>
> 1
> 10
>
>
> Is all above can be possible? 90laks datas and 1laks per day writes and
> 30laks per day read?? - if yes what type of system configuration would require.
>
> Please suggest us.
>
> thanks,
> Kalidoss.m,
>
>
> Get your world in your inbox!
>
> Mail, widgets, documents, spreadsheets, organizer and much more with your
> Sifymail WIYI id!
> Log on to http://www.sify.com
>
> ********** DISCLAIMER **********
> Information contained and transmitted by this E-MAIL is proprietary to Sify
> Limited and is intended for use only by the individual or entity to which it is
> addressed, and may contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If this is a forwarded message, the
> content of this E-MAIL may not have been sent with the authority of the Company.
> If you are not the intended recipient, an agent of the intended recipient or a
> person responsible for delivering the information to the named recipient, you
> are notified that any use, distribution, transmission, printing, copying or
> dissemination of this information in any way or in any manner is strictly
> prohibited. If you have received this communication in error, please delete this
> mail & notify us immediately at admin@sifycorp.com
Re: Solr - Load Increasing.
Posted by kalidoss <ka...@sifycorp.com>.
Thank u all.
I have increased the heap size memory from 1gb to 1.5gb. Now its
java -Xms512M -Xmx1536M -jar start.jar, My cpu load is normal and solr
is not restating frequently,
My autocommit maxdoc increased to 200.
For last 24 hours no issue on load/restarts.
Thanks Guys.
Kalidoss.m,
Otis Gospodnetic wrote:
> Your autocommit settings are still pretty aggressive causing very frequent commits, and that is using your CPU.
> Yes, splitting the servers into a master and slaves tends to be the performant/scalable way to go. There is no real downside to replication, really, just a bit of network traffic.
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: kalidoss <ka...@sifycorp.com>
>> To: solr-user@lucene.apache.org
>> Sent: Wed, November 18, 2009 2:25:05 AM
>> Subject: Re: Solr - Load Increasing.
>>
>> There seems to be some improvement. The writes speeds are faster. Server
>> restarts are lower.
>>
>> We changed the configuration to:
>> 50
>> 10000
>>
>> Before the Change:
>> - Server Restarts: 10 times in 12 hours
>> - CPU load: Average:50 and Peak:90
>>
>> After the Change:
>> - Server Restarts: 4 times in 12 hours.
>> - CPU load: Average:30 and Peak:~70
>>
>> Our every day writes are around 60k and reads are around 1 million.
>>
>> We are now changing the MaxDocs to 300 and MaxTime will be 10000 ms and
>> hoping to some more improvements.
>>
>> The system configuration is 4GB RAM and 4 core x 2 CPUs. We start the
>> solr (1.3) like this: java -Xms512M -Xmx1024M -jar start.jar
>>
>> Is there any other way we can reduce the high CPU load in the system?
>>
>> Do you guys think that upgrading to 1.4 and having the replication in
>> place with reads and writes split into separate solrs will help? How
>> efficient will the replication be with above mentioned scenarios? Is
>> there any place we can look at for info on the disadvantages of
>> replication...
>>
>> Please help.
>> Kalidoss.m,
>> Tom Alt wrote:
>>> Nice to learn a new word for the day!
>>>
>>> But to answer your question, or at least part of it, I don't really think
>>> you want a configuration like
>>>
>>>
>>> 1
>>> 10
>>>
>>>
>>> Committing every doc, and every 10 milliseconds? That's just asking for
>>> problems. How about starting with 1000 docs, and five minutes for maxTime
>>> (5*60*1000) or about 3 laks of milliseconds.
>>>
>>> That should help performance a lot. Try that, and see how it works.
>>>
>>> Tom
>>>
>>> On Mon, Nov 16, 2009 at 2:43 PM, Shashi Kant wrote:
>>>
>>>
>>>> I think it would be useful for members of this list to realize that not
>>>> everyone uses the same metrology and terms.
>>>>
>>>> It is very easy for "Americans" to use the imperial system and presume
>>>> everyone does the same; Europeans to use the metric system etc. Hopefully
>>>> members on this list would be persuaded to use or at least clarify their
>>>> terminology.
>>>>
>>>> While the apocryphal saying goes " the great thing about standards is they
>>>> are so many choose from", we should all make an effort to communicate
>>>> across
>>>> cultures and nations.
>>>>
>>>>
>>>>
>>>> On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo wrote:
>>>>
>>>>
>>>>> On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Probably "lakh": 100,000.
>>>>>>
>>>>>> So, 900k qpd and 3M docs.
>>>>>>
>>>>>> http://en.wikipedia.org/wiki/Lakh
>>>>>>
>>>>>> wunder
>>>>>>
>>>>>> On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Your autoCommit settings are very aggressive. I'm guessing that's
>>>>>>>
>>>>> what's
>>>>>
>>>>>> causing the CPU load.
>>>>>>
>>>>>>> btw. what is "laks"?
>>>>>>>
>>>>>>> Otis
>>>>>>> --
>>>>>>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>>>>>>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message ----
>>>>>>>
>>>>>>>> From: kalidoss
>>>>>>>> To: solr-user@lucene.apache.org
>>>>>>>> Sent: Mon, November 16, 2009 9:11:21 AM
>>>>>>>> Subject: Solr - Load Increasing.
>>>>>>>>
>>>>>>>> Hi All.
>>>>>>>>
>>>>>>>> My server solr box cpu utilization increasing b/w 60 to 90% and
>>>>>>>>
>>>>> some
>>>>>
>>>>>> time
>>>>>>
>>>>>>>> solr is getting down and we are restarting it manually.
>>>>>>>>
>>>>>>>> No of documents in solr 30 laks.
>>>>>>>> No of add/update requrest solr 30 thousand / day. Avg of every 30
>>>>>>>>
>>>>>> minutes
>>>>>>
>>>>>>>> around 500 writes.
>>>>>>>> No of search request 9laks / day.
>>>>>>>> Size of the data directory: 4gb.
>>>>>>>>
>>>>>>>>
>>>>>>>> My system ram is 8gb.
>>>>>>>> System available space 12gb.
>>>>>>>> processor Family: Pentium Pro
>>>>>>>>
>>>>>>>> Our solr data size can be increase in number like 90 laks. and
>>>>>>>>
>>>>> writes
>>>>>
>>>>>> per day
>>>>>>
>>>>>>>> will be around 1laks. - Hope its possible by solr.
>>>>>>>>
>>>>>>>> For write commit i have configured like
>>>>>>>>
>>>>>>>> 1
>>>>>>>> 10
>>>>>>>>
>>>>>>>>
>>>>>>>> Is all above can be possible? 90laks datas and 1laks per day
>>>>>>>>
>>>> writes
>>>>
>>>>>> and
>>>>>>
>>>>>>>> 30laks per day read?? - if yes what type of system configuration
>>>>>>>>
>>>>> would
>>>>>
>>>>>> require.
>>>>>>
>>>>>>>> Please suggest us.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Kalidoss.m,
>>>>>>>>
>>>>>>>>
>>>>>>>> Get your world in your inbox!
>>>>>>>>
>>>>>>>> Mail, widgets, documents, spreadsheets, organizer and much more with
>>>>>>>>
>>>>>> your
>>>>>>
>>>>>>>> Sifymail WIYI id!
>>>>>>>> Log on to http://www.sify.com
>>>>>>>>
>>>>>>>> ********** DISCLAIMER **********
>>>>>>>> Information contained and transmitted by this E-MAIL is proprietary
>>>>>>>>
>>>> to
>>>>
>>>>>> Sify
>>>>>>
>>>>>>>> Limited and is intended for use only by the individual or entity to
>>>>>>>>
>>>>>> which it is
>>>>>>
>>>>>>>> addressed, and may contain information that is privileged,
>>>>>>>>
>>>>> confidential
>>>>>
>>>>>> or
>>>>>>
>>>>>>>> exempt from disclosure under applicable law. If this is a forwarded
>>>>>>>>
>>>>>> message, the
>>>>>>
>>>>>>>> content of this E-MAIL may not have been sent with the authority of
>>>>>>>>
>>>>> the
>>>>>
>>>>>> Company.
>>>>>>
>>>>>>>> If you are not the intended recipient, an agent of the intended
>>>>>>>>
>>>>>> recipient or a
>>>>>>
>>>>>>>> person responsible for delivering the information to the named
>>>>>>>>
>>>>>> recipient, you
>>>>>>
>>>>>>>> are notified that any use, distribution, transmission, printing,
>>>>>>>>
>>>>> copying
>>>>>
>>>>>> or
>>>>>>
>>>>>>>> dissemination of this information in any way or in any manner is
>>>>>>>>
>>>>>> strictly
>>>>>>
>>>>>>>> prohibited. If you have received this communication in error, please
>>>>>>>>
>>>>>> delete this
>>>>>>
>>>>>>>> mail & notify us immediately at admin@sifycorp.com
>>>>>>>>
>>>>>>
>>>>> Thanks Walter for clarifying that.
>>>>>
>>>>> I too was wondering what "laks" meant.
>>>>>
>>>>> It was a bit distracting when I read the original post.
>>>>> --
>>>>> "Good Enough" is not good enough.
>>>>> To give anything less than your best is to sacrifice the gift.
>>>>> Quality First. Measure Twice. Cut Once.
>>>>>
>>>>>
>>>
>
>
Re: Solr - Load Increasing.
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Your autocommit settings are still pretty aggressive causing very frequent commits, and that is using your CPU.
Yes, splitting the servers into a master and slaves tends to be the performant/scalable way to go. There is no real downside to replication, really, just a bit of network traffic.
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: kalidoss <ka...@sifycorp.com>
> To: solr-user@lucene.apache.org
> Sent: Wed, November 18, 2009 2:25:05 AM
> Subject: Re: Solr - Load Increasing.
>
> There seems to be some improvement. The writes speeds are faster. Server
> restarts are lower.
>
> We changed the configuration to:
> 50
> 10000
>
> Before the Change:
> - Server Restarts: 10 times in 12 hours
> - CPU load: Average:50 and Peak:90
>
> After the Change:
> - Server Restarts: 4 times in 12 hours.
> - CPU load: Average:30 and Peak:~70
>
> Our every day writes are around 60k and reads are around 1 million.
>
> We are now changing the MaxDocs to 300 and MaxTime will be 10000 ms and
> hoping to some more improvements.
>
> The system configuration is 4GB RAM and 4 core x 2 CPUs. We start the
> solr (1.3) like this: java -Xms512M -Xmx1024M -jar start.jar
>
> Is there any other way we can reduce the high CPU load in the system?
>
> Do you guys think that upgrading to 1.4 and having the replication in
> place with reads and writes split into separate solrs will help? How
> efficient will the replication be with above mentioned scenarios? Is
> there any place we can look at for info on the disadvantages of
> replication...
>
> Please help.
> Kalidoss.m,
> Tom Alt wrote:
> > Nice to learn a new word for the day!
> >
> > But to answer your question, or at least part of it, I don't really think
> > you want a configuration like
> >
> >
> > 1
> > 10
> >
> >
> > Committing every doc, and every 10 milliseconds? That's just asking for
> > problems. How about starting with 1000 docs, and five minutes for maxTime
> > (5*60*1000) or about 3 laks of milliseconds.
> >
> > That should help performance a lot. Try that, and see how it works.
> >
> > Tom
> >
> > On Mon, Nov 16, 2009 at 2:43 PM, Shashi Kant wrote:
> >
> >
> >> I think it would be useful for members of this list to realize that not
> >> everyone uses the same metrology and terms.
> >>
> >> It is very easy for "Americans" to use the imperial system and presume
> >> everyone does the same; Europeans to use the metric system etc. Hopefully
> >> members on this list would be persuaded to use or at least clarify their
> >> terminology.
> >>
> >> While the apocryphal saying goes " the great thing about standards is they
> >> are so many choose from", we should all make an effort to communicate
> >> across
> >> cultures and nations.
> >>
> >>
> >>
> >> On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo wrote:
> >>
> >>
> >>> On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood
> >>>
> >>>> wrote:
> >>>>
> >>>> Probably "lakh": 100,000.
> >>>>
> >>>> So, 900k qpd and 3M docs.
> >>>>
> >>>> http://en.wikipedia.org/wiki/Lakh
> >>>>
> >>>> wunder
> >>>>
> >>>> On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
> >>>>
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Your autoCommit settings are very aggressive. I'm guessing that's
> >>>>>
> >>> what's
> >>>
> >>>> causing the CPU load.
> >>>>
> >>>>> btw. what is "laks"?
> >>>>>
> >>>>> Otis
> >>>>> --
> >>>>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> >>>>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> >>>>>
> >>>>>
> >>>>>
> >>>>> ----- Original Message ----
> >>>>>
> >>>>>> From: kalidoss
> >>>>>> To: solr-user@lucene.apache.org
> >>>>>> Sent: Mon, November 16, 2009 9:11:21 AM
> >>>>>> Subject: Solr - Load Increasing.
> >>>>>>
> >>>>>> Hi All.
> >>>>>>
> >>>>>> My server solr box cpu utilization increasing b/w 60 to 90% and
> >>>>>>
> >>> some
> >>>
> >>>> time
> >>>>
> >>>>>> solr is getting down and we are restarting it manually.
> >>>>>>
> >>>>>> No of documents in solr 30 laks.
> >>>>>> No of add/update requrest solr 30 thousand / day. Avg of every 30
> >>>>>>
> >>>> minutes
> >>>>
> >>>>>> around 500 writes.
> >>>>>> No of search request 9laks / day.
> >>>>>> Size of the data directory: 4gb.
> >>>>>>
> >>>>>>
> >>>>>> My system ram is 8gb.
> >>>>>> System available space 12gb.
> >>>>>> processor Family: Pentium Pro
> >>>>>>
> >>>>>> Our solr data size can be increase in number like 90 laks. and
> >>>>>>
> >>> writes
> >>>
> >>>> per day
> >>>>
> >>>>>> will be around 1laks. - Hope its possible by solr.
> >>>>>>
> >>>>>> For write commit i have configured like
> >>>>>>
> >>>>>> 1
> >>>>>> 10
> >>>>>>
> >>>>>>
> >>>>>> Is all above can be possible? 90laks datas and 1laks per day
> >>>>>>
> >> writes
> >>
> >>>> and
> >>>>
> >>>>>> 30laks per day read?? - if yes what type of system configuration
> >>>>>>
> >>> would
> >>>
> >>>> require.
> >>>>
> >>>>>> Please suggest us.
> >>>>>>
> >>>>>> thanks,
> >>>>>> Kalidoss.m,
> >>>>>>
> >>>>>>
> >>>>>> Get your world in your inbox!
> >>>>>>
> >>>>>> Mail, widgets, documents, spreadsheets, organizer and much more with
> >>>>>>
> >>>> your
> >>>>
> >>>>>> Sifymail WIYI id!
> >>>>>> Log on to http://www.sify.com
> >>>>>>
> >>>>>> ********** DISCLAIMER **********
> >>>>>> Information contained and transmitted by this E-MAIL is proprietary
> >>>>>>
> >> to
> >>
> >>>> Sify
> >>>>
> >>>>>> Limited and is intended for use only by the individual or entity to
> >>>>>>
> >>>> which it is
> >>>>
> >>>>>> addressed, and may contain information that is privileged,
> >>>>>>
> >>> confidential
> >>>
> >>>> or
> >>>>
> >>>>>> exempt from disclosure under applicable law. If this is a forwarded
> >>>>>>
> >>>> message, the
> >>>>
> >>>>>> content of this E-MAIL may not have been sent with the authority of
> >>>>>>
> >>> the
> >>>
> >>>> Company.
> >>>>
> >>>>>> If you are not the intended recipient, an agent of the intended
> >>>>>>
> >>>> recipient or a
> >>>>
> >>>>>> person responsible for delivering the information to the named
> >>>>>>
> >>>> recipient, you
> >>>>
> >>>>>> are notified that any use, distribution, transmission, printing,
> >>>>>>
> >>> copying
> >>>
> >>>> or
> >>>>
> >>>>>> dissemination of this information in any way or in any manner is
> >>>>>>
> >>>> strictly
> >>>>
> >>>>>> prohibited. If you have received this communication in error, please
> >>>>>>
> >>>> delete this
> >>>>
> >>>>>> mail & notify us immediately at admin@sifycorp.com
> >>>>>>
> >>>>
> >>> Thanks Walter for clarifying that.
> >>>
> >>> I too was wondering what "laks" meant.
> >>>
> >>> It was a bit distracting when I read the original post.
> >>> --
> >>> "Good Enough" is not good enough.
> >>> To give anything less than your best is to sacrifice the gift.
> >>> Quality First. Measure Twice. Cut Once.
> >>>
> >>>
> >
> >
Re: Solr - Load Increasing.
Posted by kalidoss <ka...@sifycorp.com>.
There seems to be some improvement. The writes speeds are faster. Server
restarts are lower.
We changed the configuration to:
<maxDocs>50</maxDocs>
<maxTime>10000</maxTime>
Before the Change:
- Server Restarts: 10 times in 12 hours
- CPU load: Average:50 and Peak:90
After the Change:
- Server Restarts: 4 times in 12 hours.
- CPU load: Average:30 and Peak:~70
Our every day writes are around 60k and reads are around 1 million.
We are now changing the MaxDocs to 300 and MaxTime will be 10000 ms and
hoping to some more improvements.
The system configuration is 4GB RAM and 4 core x 2 CPUs. We start the
solr (1.3) like this: java -Xms512M -Xmx1024M -jar start.jar
Is there any other way we can reduce the high CPU load in the system?
Do you guys think that upgrading to 1.4 and having the replication in
place with reads and writes split into separate solrs will help? How
efficient will the replication be with above mentioned scenarios? Is
there any place we can look at for info on the disadvantages of
replication...
Please help.
Kalidoss.m,
Tom Alt wrote:
> Nice to learn a new word for the day!
>
> But to answer your question, or at least part of it, I don't really think
> you want a configuration like
>
> <autoCommit>
> <maxDocs>1</maxDocs>
> <maxTime>10</maxTime>
> </autoCommit>
>
> Committing every doc, and every 10 milliseconds? That's just asking for
> problems. How about starting with 1000 docs, and five minutes for maxTime
> (5*60*1000) or about 3 laks of milliseconds.
>
> That should help performance a lot. Try that, and see how it works.
>
> Tom
>
> On Mon, Nov 16, 2009 at 2:43 PM, Shashi Kant <sk...@sloan.mit.edu> wrote:
>
>
>> I think it would be useful for members of this list to realize that not
>> everyone uses the same metrology and terms.
>>
>> It is very easy for "Americans" to use the imperial system and presume
>> everyone does the same; Europeans to use the metric system etc. Hopefully
>> members on this list would be persuaded to use or at least clarify their
>> terminology.
>>
>> While the apocryphal saying goes " the great thing about standards is they
>> are so many choose from", we should all make an effort to communicate
>> across
>> cultures and nations.
>>
>>
>>
>> On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo <is...@gmail.com> wrote:
>>
>>
>>> On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood <wunder@wunderwood.org
>>>
>>>> wrote:
>>>>
>>>> Probably "lakh": 100,000.
>>>>
>>>> So, 900k qpd and 3M docs.
>>>>
>>>> http://en.wikipedia.org/wiki/Lakh
>>>>
>>>> wunder
>>>>
>>>> On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> Your autoCommit settings are very aggressive. I'm guessing that's
>>>>>
>>> what's
>>>
>>>> causing the CPU load.
>>>>
>>>>> btw. what is "laks"?
>>>>>
>>>>> Otis
>>>>> --
>>>>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>>>>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message ----
>>>>>
>>>>>> From: kalidoss <ka...@sifycorp.com>
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Sent: Mon, November 16, 2009 9:11:21 AM
>>>>>> Subject: Solr - Load Increasing.
>>>>>>
>>>>>> Hi All.
>>>>>>
>>>>>> My server solr box cpu utilization increasing b/w 60 to 90% and
>>>>>>
>>> some
>>>
>>>> time
>>>>
>>>>>> solr is getting down and we are restarting it manually.
>>>>>>
>>>>>> No of documents in solr 30 laks.
>>>>>> No of add/update requrest solr 30 thousand / day. Avg of every 30
>>>>>>
>>>> minutes
>>>>
>>>>>> around 500 writes.
>>>>>> No of search request 9laks / day.
>>>>>> Size of the data directory: 4gb.
>>>>>>
>>>>>>
>>>>>> My system ram is 8gb.
>>>>>> System available space 12gb.
>>>>>> processor Family: Pentium Pro
>>>>>>
>>>>>> Our solr data size can be increase in number like 90 laks. and
>>>>>>
>>> writes
>>>
>>>> per day
>>>>
>>>>>> will be around 1laks. - Hope its possible by solr.
>>>>>>
>>>>>> For write commit i have configured like
>>>>>>
>>>>>> 1
>>>>>> 10
>>>>>>
>>>>>>
>>>>>> Is all above can be possible? 90laks datas and 1laks per day
>>>>>>
>> writes
>>
>>>> and
>>>>
>>>>>> 30laks per day read?? - if yes what type of system configuration
>>>>>>
>>> would
>>>
>>>> require.
>>>>
>>>>>> Please suggest us.
>>>>>>
>>>>>> thanks,
>>>>>> Kalidoss.m,
>>>>>>
>>>>>>
>>>>>> Get your world in your inbox!
>>>>>>
>>>>>> Mail, widgets, documents, spreadsheets, organizer and much more with
>>>>>>
>>>> your
>>>>
>>>>>> Sifymail WIYI id!
>>>>>> Log on to http://www.sify.com
>>>>>>
>>>>>> ********** DISCLAIMER **********
>>>>>> Information contained and transmitted by this E-MAIL is proprietary
>>>>>>
>> to
>>
>>>> Sify
>>>>
>>>>>> Limited and is intended for use only by the individual or entity to
>>>>>>
>>>> which it is
>>>>
>>>>>> addressed, and may contain information that is privileged,
>>>>>>
>>> confidential
>>>
>>>> or
>>>>
>>>>>> exempt from disclosure under applicable law. If this is a forwarded
>>>>>>
>>>> message, the
>>>>
>>>>>> content of this E-MAIL may not have been sent with the authority of
>>>>>>
>>> the
>>>
>>>> Company.
>>>>
>>>>>> If you are not the intended recipient, an agent of the intended
>>>>>>
>>>> recipient or a
>>>>
>>>>>> person responsible for delivering the information to the named
>>>>>>
>>>> recipient, you
>>>>
>>>>>> are notified that any use, distribution, transmission, printing,
>>>>>>
>>> copying
>>>
>>>> or
>>>>
>>>>>> dissemination of this information in any way or in any manner is
>>>>>>
>>>> strictly
>>>>
>>>>>> prohibited. If you have received this communication in error, please
>>>>>>
>>>> delete this
>>>>
>>>>>> mail & notify us immediately at admin@sifycorp.com
>>>>>>
>>>>
>>> Thanks Walter for clarifying that.
>>>
>>> I too was wondering what "laks" meant.
>>>
>>> It was a bit distracting when I read the original post.
>>> --
>>> "Good Enough" is not good enough.
>>> To give anything less than your best is to sacrifice the gift.
>>> Quality First. Measure Twice. Cut Once.
>>>
>>>
>
>
Re: Solr - Load Increasing.
Posted by Tom Alt <so...@worldware.com>.
Nice to learn a new word for the day!
But to answer your question, or at least part of it, I don't really think
you want a configuration like
<autoCommit>
<maxDocs>1</maxDocs>
<maxTime>10</maxTime>
</autoCommit>
Committing every doc, and every 10 milliseconds? That's just asking for
problems. How about starting with 1000 docs, and five minutes for maxTime
(5*60*1000) or about 3 laks of milliseconds.
That should help performance a lot. Try that, and see how it works.
Tom
On Mon, Nov 16, 2009 at 2:43 PM, Shashi Kant <sk...@sloan.mit.edu> wrote:
> I think it would be useful for members of this list to realize that not
> everyone uses the same metrology and terms.
>
> It is very easy for "Americans" to use the imperial system and presume
> everyone does the same; Europeans to use the metric system etc. Hopefully
> members on this list would be persuaded to use or at least clarify their
> terminology.
>
> While the apocryphal saying goes " the great thing about standards is they
> are so many choose from", we should all make an effort to communicate
> across
> cultures and nations.
>
>
>
> On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo <is...@gmail.com> wrote:
>
> > On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood <wunder@wunderwood.org
> > >wrote:
> >
> > > Probably "lakh": 100,000.
> > >
> > > So, 900k qpd and 3M docs.
> > >
> > > http://en.wikipedia.org/wiki/Lakh
> > >
> > > wunder
> > >
> > > On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
> > >
> > > > Hi,
> > > >
> > > > Your autoCommit settings are very aggressive. I'm guessing that's
> > what's
> > > causing the CPU load.
> > > >
> > > > btw. what is "laks"?
> > > >
> > > > Otis
> > > > --
> > > > Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> > > > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > >> From: kalidoss <ka...@sifycorp.com>
> > > >> To: solr-user@lucene.apache.org
> > > >> Sent: Mon, November 16, 2009 9:11:21 AM
> > > >> Subject: Solr - Load Increasing.
> > > >>
> > > >> Hi All.
> > > >>
> > > >> My server solr box cpu utilization increasing b/w 60 to 90% and
> > some
> > > time
> > > >> solr is getting down and we are restarting it manually.
> > > >>
> > > >> No of documents in solr 30 laks.
> > > >> No of add/update requrest solr 30 thousand / day. Avg of every 30
> > > minutes
> > > >> around 500 writes.
> > > >> No of search request 9laks / day.
> > > >> Size of the data directory: 4gb.
> > > >>
> > > >>
> > > >> My system ram is 8gb.
> > > >> System available space 12gb.
> > > >> processor Family: Pentium Pro
> > > >>
> > > >> Our solr data size can be increase in number like 90 laks. and
> > writes
> > > per day
> > > >> will be around 1laks. - Hope its possible by solr.
> > > >>
> > > >> For write commit i have configured like
> > > >>
> > > >> 1
> > > >> 10
> > > >>
> > > >>
> > > >> Is all above can be possible? 90laks datas and 1laks per day
> writes
> > > and
> > > >> 30laks per day read?? - if yes what type of system configuration
> > would
> > > require.
> > > >>
> > > >> Please suggest us.
> > > >>
> > > >> thanks,
> > > >> Kalidoss.m,
> > > >>
> > > >>
> > > >> Get your world in your inbox!
> > > >>
> > > >> Mail, widgets, documents, spreadsheets, organizer and much more with
> > > your
> > > >> Sifymail WIYI id!
> > > >> Log on to http://www.sify.com
> > > >>
> > > >> ********** DISCLAIMER **********
> > > >> Information contained and transmitted by this E-MAIL is proprietary
> to
> > > Sify
> > > >> Limited and is intended for use only by the individual or entity to
> > > which it is
> > > >> addressed, and may contain information that is privileged,
> > confidential
> > > or
> > > >> exempt from disclosure under applicable law. If this is a forwarded
> > > message, the
> > > >> content of this E-MAIL may not have been sent with the authority of
> > the
> > > Company.
> > > >> If you are not the intended recipient, an agent of the intended
> > > recipient or a
> > > >> person responsible for delivering the information to the named
> > > recipient, you
> > > >> are notified that any use, distribution, transmission, printing,
> > copying
> > > or
> > > >> dissemination of this information in any way or in any manner is
> > > strictly
> > > >> prohibited. If you have received this communication in error, please
> > > delete this
> > > >> mail & notify us immediately at admin@sifycorp.com
> > > >
> > >
> > >
> >
> >
> > Thanks Walter for clarifying that.
> >
> > I too was wondering what "laks" meant.
> >
> > It was a bit distracting when I read the original post.
> > --
> > "Good Enough" is not good enough.
> > To give anything less than your best is to sacrifice the gift.
> > Quality First. Measure Twice. Cut Once.
> >
>
Re: Solr - Load Increasing.
Posted by Shashi Kant <sk...@sloan.mit.edu>.
I think it would be useful for members of this list to realize that not
everyone uses the same metrology and terms.
It is very easy for "Americans" to use the imperial system and presume
everyone does the same; Europeans to use the metric system etc. Hopefully
members on this list would be persuaded to use or at least clarify their
terminology.
While the apocryphal saying goes " the great thing about standards is they
are so many choose from", we should all make an effort to communicate across
cultures and nations.
On Mon, Nov 16, 2009 at 5:33 PM, Israel Ekpo <is...@gmail.com> wrote:
> On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood <wunder@wunderwood.org
> >wrote:
>
> > Probably "lakh": 100,000.
> >
> > So, 900k qpd and 3M docs.
> >
> > http://en.wikipedia.org/wiki/Lakh
> >
> > wunder
> >
> > On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
> >
> > > Hi,
> > >
> > > Your autoCommit settings are very aggressive. I'm guessing that's
> what's
> > causing the CPU load.
> > >
> > > btw. what is "laks"?
> > >
> > > Otis
> > > --
> > > Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> > > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> > >
> > >
> > >
> > > ----- Original Message ----
> > >> From: kalidoss <ka...@sifycorp.com>
> > >> To: solr-user@lucene.apache.org
> > >> Sent: Mon, November 16, 2009 9:11:21 AM
> > >> Subject: Solr - Load Increasing.
> > >>
> > >> Hi All.
> > >>
> > >> My server solr box cpu utilization increasing b/w 60 to 90% and
> some
> > time
> > >> solr is getting down and we are restarting it manually.
> > >>
> > >> No of documents in solr 30 laks.
> > >> No of add/update requrest solr 30 thousand / day. Avg of every 30
> > minutes
> > >> around 500 writes.
> > >> No of search request 9laks / day.
> > >> Size of the data directory: 4gb.
> > >>
> > >>
> > >> My system ram is 8gb.
> > >> System available space 12gb.
> > >> processor Family: Pentium Pro
> > >>
> > >> Our solr data size can be increase in number like 90 laks. and
> writes
> > per day
> > >> will be around 1laks. - Hope its possible by solr.
> > >>
> > >> For write commit i have configured like
> > >>
> > >> 1
> > >> 10
> > >>
> > >>
> > >> Is all above can be possible? 90laks datas and 1laks per day writes
> > and
> > >> 30laks per day read?? - if yes what type of system configuration
> would
> > require.
> > >>
> > >> Please suggest us.
> > >>
> > >> thanks,
> > >> Kalidoss.m,
> > >>
> > >>
> > >> Get your world in your inbox!
> > >>
> > >> Mail, widgets, documents, spreadsheets, organizer and much more with
> > your
> > >> Sifymail WIYI id!
> > >> Log on to http://www.sify.com
> > >>
> > >> ********** DISCLAIMER **********
> > >> Information contained and transmitted by this E-MAIL is proprietary to
> > Sify
> > >> Limited and is intended for use only by the individual or entity to
> > which it is
> > >> addressed, and may contain information that is privileged,
> confidential
> > or
> > >> exempt from disclosure under applicable law. If this is a forwarded
> > message, the
> > >> content of this E-MAIL may not have been sent with the authority of
> the
> > Company.
> > >> If you are not the intended recipient, an agent of the intended
> > recipient or a
> > >> person responsible for delivering the information to the named
> > recipient, you
> > >> are notified that any use, distribution, transmission, printing,
> copying
> > or
> > >> dissemination of this information in any way or in any manner is
> > strictly
> > >> prohibited. If you have received this communication in error, please
> > delete this
> > >> mail & notify us immediately at admin@sifycorp.com
> > >
> >
> >
>
>
> Thanks Walter for clarifying that.
>
> I too was wondering what "laks" meant.
>
> It was a bit distracting when I read the original post.
> --
> "Good Enough" is not good enough.
> To give anything less than your best is to sacrifice the gift.
> Quality First. Measure Twice. Cut Once.
>
Re: Solr - Load Increasing.
Posted by Israel Ekpo <is...@gmail.com>.
On Mon, Nov 16, 2009 at 5:22 PM, Walter Underwood <wu...@wunderwood.org>wrote:
> Probably "lakh": 100,000.
>
> So, 900k qpd and 3M docs.
>
> http://en.wikipedia.org/wiki/Lakh
>
> wunder
>
> On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
>
> > Hi,
> >
> > Your autoCommit settings are very aggressive. I'm guessing that's what's
> causing the CPU load.
> >
> > btw. what is "laks"?
> >
> > Otis
> > --
> > Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> >
> >
> >
> > ----- Original Message ----
> >> From: kalidoss <ka...@sifycorp.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Mon, November 16, 2009 9:11:21 AM
> >> Subject: Solr - Load Increasing.
> >>
> >> Hi All.
> >>
> >> My server solr box cpu utilization increasing b/w 60 to 90% and some
> time
> >> solr is getting down and we are restarting it manually.
> >>
> >> No of documents in solr 30 laks.
> >> No of add/update requrest solr 30 thousand / day. Avg of every 30
> minutes
> >> around 500 writes.
> >> No of search request 9laks / day.
> >> Size of the data directory: 4gb.
> >>
> >>
> >> My system ram is 8gb.
> >> System available space 12gb.
> >> processor Family: Pentium Pro
> >>
> >> Our solr data size can be increase in number like 90 laks. and writes
> per day
> >> will be around 1laks. - Hope its possible by solr.
> >>
> >> For write commit i have configured like
> >>
> >> 1
> >> 10
> >>
> >>
> >> Is all above can be possible? 90laks datas and 1laks per day writes
> and
> >> 30laks per day read?? - if yes what type of system configuration would
> require.
> >>
> >> Please suggest us.
> >>
> >> thanks,
> >> Kalidoss.m,
> >>
> >>
> >> Get your world in your inbox!
> >>
> >> Mail, widgets, documents, spreadsheets, organizer and much more with
> your
> >> Sifymail WIYI id!
> >> Log on to http://www.sify.com
> >>
> >> ********** DISCLAIMER **********
> >> Information contained and transmitted by this E-MAIL is proprietary to
> Sify
> >> Limited and is intended for use only by the individual or entity to
> which it is
> >> addressed, and may contain information that is privileged, confidential
> or
> >> exempt from disclosure under applicable law. If this is a forwarded
> message, the
> >> content of this E-MAIL may not have been sent with the authority of the
> Company.
> >> If you are not the intended recipient, an agent of the intended
> recipient or a
> >> person responsible for delivering the information to the named
> recipient, you
> >> are notified that any use, distribution, transmission, printing, copying
> or
> >> dissemination of this information in any way or in any manner is
> strictly
> >> prohibited. If you have received this communication in error, please
> delete this
> >> mail & notify us immediately at admin@sifycorp.com
> >
>
>
Thanks Walter for clarifying that.
I too was wondering what "laks" meant.
It was a bit distracting when I read the original post.
--
"Good Enough" is not good enough.
To give anything less than your best is to sacrifice the gift.
Quality First. Measure Twice. Cut Once.
RE: Solr - Load Increasing.
Posted by "Sudarsan, Sithu D." <Si...@fda.hhs.gov>.
Hi,
Lakh or Lac - 100,000
Crore - 100,00,000 (ten million)
Commonly used in India
Sincerely,
Sithu D Sudarsan
-----Original Message-----
From: Walter Underwood [mailto:wunder@wunderwood.org]
Sent: Monday, November 16, 2009 5:22 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr - Load Increasing.
Probably "lakh": 100,000.
So, 900k qpd and 3M docs.
http://en.wikipedia.org/wiki/Lakh
wunder
On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
> Hi,
>
> Your autoCommit settings are very aggressive. I'm guessing that's
what's causing the CPU load.
>
> btw. what is "laks"?
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: kalidoss <ka...@sifycorp.com>
>> To: solr-user@lucene.apache.org
>> Sent: Mon, November 16, 2009 9:11:21 AM
>> Subject: Solr - Load Increasing.
>>
>> Hi All.
>>
>> My server solr box cpu utilization increasing b/w 60 to 90% and
some time
>> solr is getting down and we are restarting it manually.
>>
>> No of documents in solr 30 laks.
>> No of add/update requrest solr 30 thousand / day. Avg of every 30
minutes
>> around 500 writes.
>> No of search request 9laks / day.
>> Size of the data directory: 4gb.
>>
>>
>> My system ram is 8gb.
>> System available space 12gb.
>> processor Family: Pentium Pro
>>
>> Our solr data size can be increase in number like 90 laks. and
writes per day
>> will be around 1laks. - Hope its possible by solr.
>>
>> For write commit i have configured like
>>
>> 1
>> 10
>>
>>
>> Is all above can be possible? 90laks datas and 1laks per day writes
and
>> 30laks per day read?? - if yes what type of system configuration
would require.
>>
>> Please suggest us.
>>
>> thanks,
>> Kalidoss.m,
>>
>>
>> Get your world in your inbox!
>>
>> Mail, widgets, documents, spreadsheets, organizer and much more with
your
>> Sifymail WIYI id!
>> Log on to http://www.sify.com
>>
>> ********** DISCLAIMER **********
>> Information contained and transmitted by this E-MAIL is proprietary
to Sify
>> Limited and is intended for use only by the individual or entity to
which it is
>> addressed, and may contain information that is privileged,
confidential or
>> exempt from disclosure under applicable law. If this is a forwarded
message, the
>> content of this E-MAIL may not have been sent with the authority of
the Company.
>> If you are not the intended recipient, an agent of the intended
recipient or a
>> person responsible for delivering the information to the named
recipient, you
>> are notified that any use, distribution, transmission, printing,
copying or
>> dissemination of this information in any way or in any manner is
strictly
>> prohibited. If you have received this communication in error, please
delete this
>> mail & notify us immediately at admin@sifycorp.com
>
Re: Solr - Load Increasing.
Posted by Walter Underwood <wu...@wunderwood.org>.
Probably "lakh": 100,000.
So, 900k qpd and 3M docs.
http://en.wikipedia.org/wiki/Lakh
wunder
On Nov 16, 2009, at 2:17 PM, Otis Gospodnetic wrote:
> Hi,
>
> Your autoCommit settings are very aggressive. I'm guessing that's what's causing the CPU load.
>
> btw. what is "laks"?
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: kalidoss <ka...@sifycorp.com>
>> To: solr-user@lucene.apache.org
>> Sent: Mon, November 16, 2009 9:11:21 AM
>> Subject: Solr - Load Increasing.
>>
>> Hi All.
>>
>> My server solr box cpu utilization increasing b/w 60 to 90% and some time
>> solr is getting down and we are restarting it manually.
>>
>> No of documents in solr 30 laks.
>> No of add/update requrest solr 30 thousand / day. Avg of every 30 minutes
>> around 500 writes.
>> No of search request 9laks / day.
>> Size of the data directory: 4gb.
>>
>>
>> My system ram is 8gb.
>> System available space 12gb.
>> processor Family: Pentium Pro
>>
>> Our solr data size can be increase in number like 90 laks. and writes per day
>> will be around 1laks. - Hope its possible by solr.
>>
>> For write commit i have configured like
>>
>> 1
>> 10
>>
>>
>> Is all above can be possible? 90laks datas and 1laks per day writes and
>> 30laks per day read?? - if yes what type of system configuration would require.
>>
>> Please suggest us.
>>
>> thanks,
>> Kalidoss.m,
>>
>>
>> Get your world in your inbox!
>>
>> Mail, widgets, documents, spreadsheets, organizer and much more with your
>> Sifymail WIYI id!
>> Log on to http://www.sify.com
>>
>> ********** DISCLAIMER **********
>> Information contained and transmitted by this E-MAIL is proprietary to Sify
>> Limited and is intended for use only by the individual or entity to which it is
>> addressed, and may contain information that is privileged, confidential or
>> exempt from disclosure under applicable law. If this is a forwarded message, the
>> content of this E-MAIL may not have been sent with the authority of the Company.
>> If you are not the intended recipient, an agent of the intended recipient or a
>> person responsible for delivering the information to the named recipient, you
>> are notified that any use, distribution, transmission, printing, copying or
>> dissemination of this information in any way or in any manner is strictly
>> prohibited. If you have received this communication in error, please delete this
>> mail & notify us immediately at admin@sifycorp.com
>