You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bruno Mannina <bm...@free.fr> on 2012/05/22 09:06:02 UTC
System requirements in my case?
Dear Solr users,
My company would like to use solr to index around 80 000 000 documents
(xml files with around 5~10ko size each).
My program (robot) will connect to this solr with boolean requests.
Number of users: around 1000
Number of requests by user and by day: 300
Number of users by day: 30
I would like to subscribe to a host provider with this configuration:
- Dedicated Server
- Ubuntu
- Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
- Unlimited bandwidth
- IP fixe
Do you think this configuration is enough?
Thanks for your info,
Sincerely
Bruno
Re: System requirements in my case?
Posted by Bruno Mannina <bm...@free.fr>.
Hi,
facets I don't know yet because I don't know exactly what is facets (sorry)
Sorting: yes
Scoring: yes
Concerning update Frequency : every week
Volume: around 1Go data by year
Merci beaucoup :)
Aix En Provence
France
Le 22/05/2012 10:35, lboutros a écrit :
> Hi Bruno,
>
> will you use facets and result sorting ?
> What is the update frequency/volume ?
>
> This could impact the amount of memory/server count.
>
> Ludovic.
>
> -----
> Jouve
> France.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/System-requirements-in-my-case-tp3985309p3985327.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: System requirements in my case?
Posted by lboutros <bo...@gmail.com>.
Hi Bruno,
will you use facets and result sorting ?
What is the update frequency/volume ?
This could impact the amount of memory/server count.
Ludovic.
-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/System-requirements-in-my-case-tp3985309p3985327.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: System requirements in my case?
Posted by Bruno Mannina <bm...@free.fr>.
Thanks a lot for all these help !
Le 24/05/2012 09:12, Otis Gospodnetic a écrit :
> Bruno,
>
> You can use jconsole to see the size of the JVM heap, if that's what you are after.
>
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm
>
>
>
>> ________________________________
>> From: Bruno Mannina<bm...@free.fr>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, May 22, 2012 8:43 AM
>> Subject: Re: System requirements in my case?
>>
>> I installed a temp server on my university with 12 000 docs (Ubuntu+solr
>> 3.6.0)
>> May be I can preview the size of memory I need?
>>
>> Q: How can I check the memory used?
>>
>>
>> Le 22/05/2012 13:14, findbestopensource a écrit :
>>> Seems to be fine. Go head.
>>>
>>> Before hosting, Have you tried / tested your application in local setup.
>>> RAM usage is what matters in terms of Solr. Just benchmark your app for 100
>>> 000 documents, Log the memory used. Calculate the RAM reqd for 80 000 000
>>> documents.
>>>
>>> Regards
>>> Aditya
>>> www.findbestopensource.com
>>>
>>>
>>> On Tue, May 22, 2012 at 2:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>>> My choice: http://www.ovh.com/fr/**serveurs_dedies/eg_best_of.xml<http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml>
>>>>
>>>> 24 Go DDR3
>>>>
>>>> Le 22/05/2012 10:26, findbestopensource a écrit :
>>>>
>>>> Dedicated Server may not be required. If you want to cut down cost, then
>>>>> prefer shared server.
>>>>>
>>>>> How much the RAM?
>>>>>
>>>>> Regards
>>>>> Aditya
>>>>> www.findbestopensource.com
>>>>>
>>>>>
>>>>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>
>>>>> Dear Solr users,
>>>>>> My company would like to use solr to index around 80 000 000 documents
>>>>>> (xml files with around 5~10ko size each).
>>>>>> My program (robot) will connect to this solr with boolean requests.
>>>>>>
>>>>>> Number of users: around 1000
>>>>>> Number of requests by user and by day: 300
>>>>>> Number of users by day: 30
>>>>>>
>>>>>> I would like to subscribe to a host provider with this configuration:
>>>>>> - Dedicated Server
>>>>>> - Ubuntu
>>>>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>>>>> - Unlimited bandwidth
>>>>>> - IP fixe
>>>>>>
>>>>>> Do you think this configuration is enough?
>>>>>>
>>>>>> Thanks for your info,
>>>>>> Sincerely
>>>>>> Bruno
>>>>>>
>>>>>>
>>
>>
>>
Re: System requirements in my case?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Bruno,
You can use jconsole to see the size of the JVM heap, if that's what you are after.
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm
>________________________________
> From: Bruno Mannina <bm...@free.fr>
>To: solr-user@lucene.apache.org
>Sent: Tuesday, May 22, 2012 8:43 AM
>Subject: Re: System requirements in my case?
>
>I installed a temp server on my university with 12 000 docs (Ubuntu+solr
>3.6.0)
>May be I can preview the size of memory I need?
>
>Q: How can I check the memory used?
>
>
>Le 22/05/2012 13:14, findbestopensource a écrit :
>> Seems to be fine. Go head.
>>
>> Before hosting, Have you tried / tested your application in local setup.
>> RAM usage is what matters in terms of Solr. Just benchmark your app for 100
>> 000 documents, Log the memory used. Calculate the RAM reqd for 80 000 000
>> documents.
>>
>> Regards
>> Aditya
>> www.findbestopensource.com
>>
>>
>> On Tue, May 22, 2012 at 2:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>
>>> My choice: http://www.ovh.com/fr/**serveurs_dedies/eg_best_of.xml<http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml>
>>>
>>> 24 Go DDR3
>>>
>>> Le 22/05/2012 10:26, findbestopensource a écrit :
>>>
>>> Dedicated Server may not be required. If you want to cut down cost, then
>>>> prefer shared server.
>>>>
>>>> How much the RAM?
>>>>
>>>> Regards
>>>> Aditya
>>>> www.findbestopensource.com
>>>>
>>>>
>>>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>>
>>>> Dear Solr users,
>>>>> My company would like to use solr to index around 80 000 000 documents
>>>>> (xml files with around 5~10ko size each).
>>>>> My program (robot) will connect to this solr with boolean requests.
>>>>>
>>>>> Number of users: around 1000
>>>>> Number of requests by user and by day: 300
>>>>> Number of users by day: 30
>>>>>
>>>>> I would like to subscribe to a host provider with this configuration:
>>>>> - Dedicated Server
>>>>> - Ubuntu
>>>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>>>> - Unlimited bandwidth
>>>>> - IP fixe
>>>>>
>>>>> Do you think this configuration is enough?
>>>>>
>>>>> Thanks for your info,
>>>>> Sincerely
>>>>> Bruno
>>>>>
>>>>>
>
>
>
>
Re: System requirements in my case?
Posted by Bruno Mannina <bm...@free.fr>.
I installed a temp server on my university with 12 000 docs (Ubuntu+solr
3.6.0)
May be I can preview the size of memory I need?
Q: How can I check the memory used?
Le 22/05/2012 13:14, findbestopensource a écrit :
> Seems to be fine. Go head.
>
> Before hosting, Have you tried / tested your application in local setup.
> RAM usage is what matters in terms of Solr. Just benchmark your app for 100
> 000 documents, Log the memory used. Calculate the RAM reqd for 80 000 000
> documents.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
> On Tue, May 22, 2012 at 2:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>
>> My choice: http://www.ovh.com/fr/**serveurs_dedies/eg_best_of.xml<http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml>
>>
>> 24 Go DDR3
>>
>> Le 22/05/2012 10:26, findbestopensource a écrit :
>>
>> Dedicated Server may not be required. If you want to cut down cost, then
>>> prefer shared server.
>>>
>>> How much the RAM?
>>>
>>> Regards
>>> Aditya
>>> www.findbestopensource.com
>>>
>>>
>>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>> Dear Solr users,
>>>> My company would like to use solr to index around 80 000 000 documents
>>>> (xml files with around 5~10ko size each).
>>>> My program (robot) will connect to this solr with boolean requests.
>>>>
>>>> Number of users: around 1000
>>>> Number of requests by user and by day: 300
>>>> Number of users by day: 30
>>>>
>>>> I would like to subscribe to a host provider with this configuration:
>>>> - Dedicated Server
>>>> - Ubuntu
>>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>>> - Unlimited bandwidth
>>>> - IP fixe
>>>>
>>>> Do you think this configuration is enough?
>>>>
>>>> Thanks for your info,
>>>> Sincerely
>>>> Bruno
>>>>
>>>>
Re: System requirements in my case?
Posted by findbestopensource <fi...@gmail.com>.
Seems to be fine. Go head.
Before hosting, Have you tried / tested your application in local setup.
RAM usage is what matters in terms of Solr. Just benchmark your app for 100
000 documents, Log the memory used. Calculate the RAM reqd for 80 000 000
documents.
Regards
Aditya
www.findbestopensource.com
On Tue, May 22, 2012 at 2:36 PM, Bruno Mannina <bm...@free.fr> wrote:
> My choice: http://www.ovh.com/fr/**serveurs_dedies/eg_best_of.xml<http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml>
>
> 24 Go DDR3
>
> Le 22/05/2012 10:26, findbestopensource a écrit :
>
> Dedicated Server may not be required. If you want to cut down cost, then
>> prefer shared server.
>>
>> How much the RAM?
>>
>> Regards
>> Aditya
>> www.findbestopensource.com
>>
>>
>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>
>> Dear Solr users,
>>>
>>> My company would like to use solr to index around 80 000 000 documents
>>> (xml files with around 5~10ko size each).
>>> My program (robot) will connect to this solr with boolean requests.
>>>
>>> Number of users: around 1000
>>> Number of requests by user and by day: 300
>>> Number of users by day: 30
>>>
>>> I would like to subscribe to a host provider with this configuration:
>>> - Dedicated Server
>>> - Ubuntu
>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>> - Unlimited bandwidth
>>> - IP fixe
>>>
>>> Do you think this configuration is enough?
>>>
>>> Thanks for your info,
>>> Sincerely
>>> Bruno
>>>
>>>
>
Re:Re: System requirements in my case?
Posted by 架子车诱惑 <ji...@126.com>.
How Remove the repetition of the ''MoreLikethis'' data ?
--79845
Re: System requirements in my case?
Posted by Jan Høydahl <ja...@cominvent.com>.
Well, 12000 is probably too little to do a representative sizing, but you can try an optimize() and then calculate what the size will be for 80mill docs. You'll definitely not be able to cache the whole index in memory on one server, but if you can live with that kind of performance then it's ok. Btw. clustering may get very expensive since you need to return many more hits than normally and that means disk I/O..
> Another question concerning the execution of solr, have just to run java -jar start.jar ?
> or you think I must run it with another way ?
First you must decide what application server to use. You may of course use the built-in Jetty server which comes with Solr if you wish. You will have to set JVM parameters for memory, garbage collection, logging etc. See http://wiki.apache.org/solr/SolrPerformanceFactors and http://wiki.apache.org/solr/SolrJetty
--
Jan Høydahl, search solution architect
Cominvent AS - www.facebook.com/Cominvent
Solr Training - www.solrtraining.com
On 22. mai 2012, at 15:16, Bruno Mannina wrote:
> Hi Jan,
>
> Thanks for all these details !
>
> Answers are below.
>
> Sincerely,
> Bruno
>
>
> Le 22/05/2012 13:58, Jan Høydahl a écrit :
>> Hi,
>>
>> It is impossible to guess the required HW size without more knowledge about data and usage. 80 mill docs is a fair amount.
>>
>> Here's how I would approach sizing the setup:
>> 1) Get your schema in shape, removing unnecessary stored/indexed fields
> Ok good idea !
>> 2) To a test index locally of a part of the dataset, e.g. 10 mill docs and perform an Optimize
> Concerning test, I have only actually a sample with 12000 docs. no more :'(
>> 3) Measure the size of the index folder, multiply with 8 to get a clue of total index size
> With 12 000 docs my index folder size is: 33Mo
> ps: I use "solr.clustering.enabled=true"
>
>> 4) Do some benchmarking with realistic types of queries to identify performance bottlenecks on query side
> yep, this point is for later.
>
>> Depending on your requirements for search performance, you can beef up your RAM to hold the whole index or depend on slow disks as a bottleneck. If you find that total size of index is 16Gb, you should leave>16Gb free for OS disk caching, e.g. allocate 8Gb to Tomcat/Solr and leave the rest for the OS. If I should guess, you probably find that one server gets overloaded or too slow with your amount of docs, and that you end up with sharding across 2-4 servers.
> I will take a look to see if I can easely increase RAM on the server (actually 24Go)
>
> Another question concerning the execution of solr, have just to run java -jar start.jar ?
> or you think I must run it with another way ?
>
>
>> PS: Do you always need to search all data? A trick may be to partition your data such that say 80% of searches go to a "fresh" index with 10% of the content, while the remaining searches include everything.
> Yes I need to search to the whole index, even old document must be requested.
>
>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.facebook.com/Cominvent
>> Solr Training - www.solrtraining.com
>>
>> On 22. mai 2012, at 11:06, Bruno Mannina wrote:
>>
>>> My choice: http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml
>>>
>>> 24 Go DDR3
>>>
>>> Le 22/05/2012 10:26, findbestopensource a écrit :
>>>> Dedicated Server may not be required. If you want to cut down cost, then
>>>> prefer shared server.
>>>>
>>>> How much the RAM?
>>>>
>>>> Regards
>>>> Aditya
>>>> www.findbestopensource.com
>>>>
>>>>
>>>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>>
>>>>> Dear Solr users,
>>>>>
>>>>> My company would like to use solr to index around 80 000 000 documents
>>>>> (xml files with around 5~10ko size each).
>>>>> My program (robot) will connect to this solr with boolean requests.
>>>>>
>>>>> Number of users: around 1000
>>>>> Number of requests by user and by day: 300
>>>>> Number of users by day: 30
>>>>>
>>>>> I would like to subscribe to a host provider with this configuration:
>>>>> - Dedicated Server
>>>>> - Ubuntu
>>>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>>>> - Unlimited bandwidth
>>>>> - IP fixe
>>>>>
>>>>> Do you think this configuration is enough?
>>>>>
>>>>> Thanks for your info,
>>>>> Sincerely
>>>>> Bruno
>>>>>
>>
>>
>
Re: System requirements in my case?
Posted by Stanislaw Osinski <st...@osinski.name>.
>
> 3) Measure the size of the index folder, multiply with 8 to get a clue of
>> total index size
>>
> With 12 000 docs my index folder size is: 33Mo
> ps: I use "solr.clustering.enabled=true"
Clustering is performed at search time, it doesn't affect the size of the
index (but obviously it does affect the search response times).
Staszek
Re: System requirements in my case?
Posted by Bruno Mannina <bm...@free.fr>.
Hi Jan,
Thanks for all these details !
Answers are below.
Sincerely,
Bruno
Le 22/05/2012 13:58, Jan Høydahl a écrit :
> Hi,
>
> It is impossible to guess the required HW size without more knowledge about data and usage. 80 mill docs is a fair amount.
>
> Here's how I would approach sizing the setup:
> 1) Get your schema in shape, removing unnecessary stored/indexed fields
Ok good idea !
> 2) To a test index locally of a part of the dataset, e.g. 10 mill docs and perform an Optimize
Concerning test, I have only actually a sample with 12000 docs. no more :'(
> 3) Measure the size of the index folder, multiply with 8 to get a clue of total index size
With 12 000 docs my index folder size is: 33Mo
ps: I use "solr.clustering.enabled=true"
> 4) Do some benchmarking with realistic types of queries to identify performance bottlenecks on query side
yep, this point is for later.
> Depending on your requirements for search performance, you can beef up your RAM to hold the whole index or depend on slow disks as a bottleneck. If you find that total size of index is 16Gb, you should leave>16Gb free for OS disk caching, e.g. allocate 8Gb to Tomcat/Solr and leave the rest for the OS. If I should guess, you probably find that one server gets overloaded or too slow with your amount of docs, and that you end up with sharding across 2-4 servers.
I will take a look to see if I can easely increase RAM on the server
(actually 24Go)
Another question concerning the execution of solr, have just to run java
-jar start.jar ?
or you think I must run it with another way ?
> PS: Do you always need to search all data? A trick may be to partition your data such that say 80% of searches go to a "fresh" index with 10% of the content, while the remaining searches include everything.
Yes I need to search to the whole index, even old document must be
requested.
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.facebook.com/Cominvent
> Solr Training - www.solrtraining.com
>
> On 22. mai 2012, at 11:06, Bruno Mannina wrote:
>
>> My choice: http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml
>>
>> 24 Go DDR3
>>
>> Le 22/05/2012 10:26, findbestopensource a écrit :
>>> Dedicated Server may not be required. If you want to cut down cost, then
>>> prefer shared server.
>>>
>>> How much the RAM?
>>>
>>> Regards
>>> Aditya
>>> www.findbestopensource.com
>>>
>>>
>>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>>> Dear Solr users,
>>>>
>>>> My company would like to use solr to index around 80 000 000 documents
>>>> (xml files with around 5~10ko size each).
>>>> My program (robot) will connect to this solr with boolean requests.
>>>>
>>>> Number of users: around 1000
>>>> Number of requests by user and by day: 300
>>>> Number of users by day: 30
>>>>
>>>> I would like to subscribe to a host provider with this configuration:
>>>> - Dedicated Server
>>>> - Ubuntu
>>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>>> - Unlimited bandwidth
>>>> - IP fixe
>>>>
>>>> Do you think this configuration is enough?
>>>>
>>>> Thanks for your info,
>>>> Sincerely
>>>> Bruno
>>>>
>
>
Re: System requirements in my case?
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
It is impossible to guess the required HW size without more knowledge about data and usage. 80 mill docs is a fair amount.
Here's how I would approach sizing the setup:
1) Get your schema in shape, removing unnecessary stored/indexed fields
2) To a test index locally of a part of the dataset, e.g. 10 mill docs and perform an Optimize
3) Measure the size of the index folder, multiply with 8 to get a clue of total index size
4) Do some benchmarking with realistic types of queries to identify performance bottlenecks on query side
Depending on your requirements for search performance, you can beef up your RAM to hold the whole index or depend on slow disks as a bottleneck. If you find that total size of index is 16Gb, you should leave >16Gb free for OS disk caching, e.g. allocate 8Gb to Tomcat/Solr and leave the rest for the OS. If I should guess, you probably find that one server gets overloaded or too slow with your amount of docs, and that you end up with sharding across 2-4 servers.
PS: Do you always need to search all data? A trick may be to partition your data such that say 80% of searches go to a "fresh" index with 10% of the content, while the remaining searches include everything.
--
Jan Høydahl, search solution architect
Cominvent AS - www.facebook.com/Cominvent
Solr Training - www.solrtraining.com
On 22. mai 2012, at 11:06, Bruno Mannina wrote:
> My choice: http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml
>
> 24 Go DDR3
>
> Le 22/05/2012 10:26, findbestopensource a écrit :
>> Dedicated Server may not be required. If you want to cut down cost, then
>> prefer shared server.
>>
>> How much the RAM?
>>
>> Regards
>> Aditya
>> www.findbestopensource.com
>>
>>
>> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>>
>>> Dear Solr users,
>>>
>>> My company would like to use solr to index around 80 000 000 documents
>>> (xml files with around 5~10ko size each).
>>> My program (robot) will connect to this solr with boolean requests.
>>>
>>> Number of users: around 1000
>>> Number of requests by user and by day: 300
>>> Number of users by day: 30
>>>
>>> I would like to subscribe to a host provider with this configuration:
>>> - Dedicated Server
>>> - Ubuntu
>>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>>> - Unlimited bandwidth
>>> - IP fixe
>>>
>>> Do you think this configuration is enough?
>>>
>>> Thanks for your info,
>>> Sincerely
>>> Bruno
>>>
>
Re: System requirements in my case?
Posted by Bruno Mannina <bm...@free.fr>.
My choice: http://www.ovh.com/fr/serveurs_dedies/eg_best_of.xml
24 Go DDR3
Le 22/05/2012 10:26, findbestopensource a écrit :
> Dedicated Server may not be required. If you want to cut down cost, then
> prefer shared server.
>
> How much the RAM?
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
> On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina<bm...@free.fr> wrote:
>
>> Dear Solr users,
>>
>> My company would like to use solr to index around 80 000 000 documents
>> (xml files with around 5~10ko size each).
>> My program (robot) will connect to this solr with boolean requests.
>>
>> Number of users: around 1000
>> Number of requests by user and by day: 300
>> Number of users by day: 30
>>
>> I would like to subscribe to a host provider with this configuration:
>> - Dedicated Server
>> - Ubuntu
>> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
>> - Unlimited bandwidth
>> - IP fixe
>>
>> Do you think this configuration is enough?
>>
>> Thanks for your info,
>> Sincerely
>> Bruno
>>
Re: System requirements in my case?
Posted by findbestopensource <fi...@gmail.com>.
Dedicated Server may not be required. If you want to cut down cost, then
prefer shared server.
How much the RAM?
Regards
Aditya
www.findbestopensource.com
On Tue, May 22, 2012 at 12:36 PM, Bruno Mannina <bm...@free.fr> wrote:
> Dear Solr users,
>
> My company would like to use solr to index around 80 000 000 documents
> (xml files with around 5~10ko size each).
> My program (robot) will connect to this solr with boolean requests.
>
> Number of users: around 1000
> Number of requests by user and by day: 300
> Number of users by day: 30
>
> I would like to subscribe to a host provider with this configuration:
> - Dedicated Server
> - Ubuntu
> - Intel Xeon i7 2x 266+ GHz 12 Go 2 * 1500Go
> - Unlimited bandwidth
> - IP fixe
>
> Do you think this configuration is enough?
>
> Thanks for your info,
> Sincerely
> Bruno
>