Performance of Solr on different Platforms

[Eswar K <kj...@gmail.com>]

Hi,

I understand that Solr can be used on different Linux flavors. Is there any
preferred flavor (Like Red Hat, Ubuntu, etc)?
Also what is the kind of configuration of hardware (Processors, RAM, etc) be
best suited for the install?
We expect to load it with millions of documents (varying from 2 - 20
million). There might be around 1000 concurrent users.

Your help in this regard will be appreciated.

Regards,
Eswar

Re: Performance of Solr on different Platforms

[Rishabh Joshi <ri...@gmail.com>]

Eswar,

This link would give you a fair idea of how Solr is used by some of the
sites/companies -
http://wiki.apache.org/solr/SolrPerformanceData

Rishabh

On Nov 20, 2007 10:49 AM, Eswar K <kj...@gmail.com> wrote:

> In our case, the load is kind of distributed. On an average, the QPS could
> be much less than that. 1000 qps could be the peak load ever expected
> could
> ever reach. However the number of documents going to be in the range of 2
> -
> 20 million documents.
>
> We would possibly distribute the indexes to different solr instances and
> possibly direct it accordingly to reduce the QPS.
>
> - Eswar
>
> On Nov 20, 2007 10:42 AM, Walter Underwood <wu...@netflix.com> wrote:
>
> > 1000 qps is a lot of load, at least 30M queries/day.
> >
> > We are running dual CPU Power P5 machines and getting about 80 qps
> > with worst case response times of 5 seconds. 90% of responses are
> > under 70 msec.
> >
> > Our expected peak load is 300 qps on our back-end Solr farm.
> > We execute multiple back-end queries for each query page.
> >
> > With N+1 sizing (full throughput with one server down), we
> > have five servers to do that. We have a separate server
> > for indexing and use the Solr distribution scripts.
> >
> > We have a relatively small index, about 250K docs.
> >
> > wunder
> >
> >
> > On 11/19/07 8:48 PM, "Eswar K" <kj...@gmail.com> wrote:
> >
> > > Its not going to hit 1000 all the time, its the expected peak value.
> > >
> > > I guess for distributing the load we should be using collections and I
> > was
> > > looking at the collections documentation (
> > > http://wiki.apache.org/solr/CollectionDistribution) .
> > >
> > > - Eswar
> > > On Nov 20, 2007 12:07 AM, Matthew Runo <mr...@zappos.com> wrote:
> > >
> > >> I'd think that any platform that can run Java would be fine to run
> > >> SOLR on. Maybe this is more a question of preferred platforms for
> Java
> > >> deployments? That is quite the load for SOLR though, you may find
> that
> > >> you want more than one server.
> > >>
> > >> Do you mean that you're expecting about 1000 QPS over an index with
> up
> > >> to 20 million documents?
> > >>
> > >> --Matthew
> > >>
> > >> On Nov 19, 2007, at 6:00 AM, Eswar K wrote:
> > >>
> > >>> All,
> > >>>
> > >>> Can you give some information on this or atleast let me know where I
> > >>> can
> > >>> find this information if its already listed out anywhere.
> > >>>
> > >>> Regards,
> > >>> Eswar
> > >>>
> > >>> On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> I understand that Solr can be used on different Linux flavors. Is
> > >>>> there
> > >>>> any preferred flavor (Like Red Hat, Ubuntu, etc)?
> > >>>> Also what is the kind of configuration of hardware (Processors,
> > >>>> RAM, etc)
> > >>>> be best suited for the install?
> > >>>> We expect to load it with millions of documents (varying from 2 -
> 20
> > >>>> million). There might be around 1000 concurrent users.
> > >>>>
> > >>>> Your help in this regard will be appreciated.
> > >>>>
> > >>>> Regards,
> > >>>> Eswar
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: Performance of Solr on different Platforms

[Eswar K <kj...@gmail.com>]

In our case, the load is kind of distributed. On an average, the QPS could
be much less than that. 1000 qps could be the peak load ever expected could
ever reach. However the number of documents going to be in the range of 2 -
20 million documents.

We would possibly distribute the indexes to different solr instances and
possibly direct it accordingly to reduce the QPS.

- Eswar

On Nov 20, 2007 10:42 AM, Walter Underwood <wu...@netflix.com> wrote:

> 1000 qps is a lot of load, at least 30M queries/day.
>
> We are running dual CPU Power P5 machines and getting about 80 qps
> with worst case response times of 5 seconds. 90% of responses are
> under 70 msec.
>
> Our expected peak load is 300 qps on our back-end Solr farm.
> We execute multiple back-end queries for each query page.
>
> With N+1 sizing (full throughput with one server down), we
> have five servers to do that. We have a separate server
> for indexing and use the Solr distribution scripts.
>
> We have a relatively small index, about 250K docs.
>
> wunder
>
>
> On 11/19/07 8:48 PM, "Eswar K" <kj...@gmail.com> wrote:
>
> > Its not going to hit 1000 all the time, its the expected peak value.
> >
> > I guess for distributing the load we should be using collections and I
> was
> > looking at the collections documentation (
> > http://wiki.apache.org/solr/CollectionDistribution) .
> >
> > - Eswar
> > On Nov 20, 2007 12:07 AM, Matthew Runo <mr...@zappos.com> wrote:
> >
> >> I'd think that any platform that can run Java would be fine to run
> >> SOLR on. Maybe this is more a question of preferred platforms for Java
> >> deployments? That is quite the load for SOLR though, you may find that
> >> you want more than one server.
> >>
> >> Do you mean that you're expecting about 1000 QPS over an index with up
> >> to 20 million documents?
> >>
> >> --Matthew
> >>
> >> On Nov 19, 2007, at 6:00 AM, Eswar K wrote:
> >>
> >>> All,
> >>>
> >>> Can you give some information on this or atleast let me know where I
> >>> can
> >>> find this information if its already listed out anywhere.
> >>>
> >>> Regards,
> >>> Eswar
> >>>
> >>> On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I understand that Solr can be used on different Linux flavors. Is
> >>>> there
> >>>> any preferred flavor (Like Red Hat, Ubuntu, etc)?
> >>>> Also what is the kind of configuration of hardware (Processors,
> >>>> RAM, etc)
> >>>> be best suited for the install?
> >>>> We expect to load it with millions of documents (varying from 2 - 20
> >>>> million). There might be around 1000 concurrent users.
> >>>>
> >>>> Your help in this regard will be appreciated.
> >>>>
> >>>> Regards,
> >>>> Eswar
> >>>>
> >>>>
> >>
> >>
>
>

Re: Performance of Solr on different Platforms

[Walter Underwood <wu...@netflix.com>]

1000 qps is a lot of load, at least 30M queries/day.

We are running dual CPU Power P5 machines and getting about 80 qps
with worst case response times of 5 seconds. 90% of responses are
under 70 msec.

Our expected peak load is 300 qps on our back-end Solr farm.
We execute multiple back-end queries for each query page.

With N+1 sizing (full throughput with one server down), we
have five servers to do that. We have a separate server
for indexing and use the Solr distribution scripts.

We have a relatively small index, about 250K docs.

wunder 


On 11/19/07 8:48 PM, "Eswar K" <kj...@gmail.com> wrote:

> Its not going to hit 1000 all the time, its the expected peak value.
> 
> I guess for distributing the load we should be using collections and I was
> looking at the collections documentation (
> http://wiki.apache.org/solr/CollectionDistribution) .
> 
> - Eswar
> On Nov 20, 2007 12:07 AM, Matthew Runo <mr...@zappos.com> wrote:
> 
>> I'd think that any platform that can run Java would be fine to run
>> SOLR on. Maybe this is more a question of preferred platforms for Java
>> deployments? That is quite the load for SOLR though, you may find that
>> you want more than one server.
>> 
>> Do you mean that you're expecting about 1000 QPS over an index with up
>> to 20 million documents?
>> 
>> --Matthew
>> 
>> On Nov 19, 2007, at 6:00 AM, Eswar K wrote:
>> 
>>> All,
>>> 
>>> Can you give some information on this or atleast let me know where I
>>> can
>>> find this information if its already listed out anywhere.
>>> 
>>> Regards,
>>> Eswar
>>> 
>>> On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I understand that Solr can be used on different Linux flavors. Is
>>>> there
>>>> any preferred flavor (Like Red Hat, Ubuntu, etc)?
>>>> Also what is the kind of configuration of hardware (Processors,
>>>> RAM, etc)
>>>> be best suited for the install?
>>>> We expect to load it with millions of documents (varying from 2 - 20
>>>> million). There might be around 1000 concurrent users.
>>>> 
>>>> Your help in this regard will be appreciated.
>>>> 
>>>> Regards,
>>>> Eswar
>>>> 
>>>> 
>> 
>> 

Re: Performance of Solr on different Platforms

[Eswar K <kj...@gmail.com>]

Its not going to hit 1000 all the time, its the expected peak value.

I guess for distributing the load we should be using collections and I was
looking at the collections documentation (
http://wiki.apache.org/solr/CollectionDistribution) .

- Eswar
On Nov 20, 2007 12:07 AM, Matthew Runo <mr...@zappos.com> wrote:

> I'd think that any platform that can run Java would be fine to run
> SOLR on. Maybe this is more a question of preferred platforms for Java
> deployments? That is quite the load for SOLR though, you may find that
> you want more than one server.
>
> Do you mean that you're expecting about 1000 QPS over an index with up
> to 20 million documents?
>
> --Matthew
>
> On Nov 19, 2007, at 6:00 AM, Eswar K wrote:
>
> > All,
> >
> > Can you give some information on this or atleast let me know where I
> > can
> > find this information if its already listed out anywhere.
> >
> > Regards,
> > Eswar
> >
> > On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I understand that Solr can be used on different Linux flavors. Is
> >> there
> >> any preferred flavor (Like Red Hat, Ubuntu, etc)?
> >> Also what is the kind of configuration of hardware (Processors,
> >> RAM, etc)
> >> be best suited for the install?
> >> We expect to load it with millions of documents (varying from 2 - 20
> >> million). There might be around 1000 concurrent users.
> >>
> >> Your help in this regard will be appreciated.
> >>
> >> Regards,
> >> Eswar
> >>
> >>
>
>

Re: Performance of Solr on different Platforms

[Matthew Runo <mr...@zappos.com>]

I'd think that any platform that can run Java would be fine to run  
SOLR on. Maybe this is more a question of preferred platforms for Java  
deployments? That is quite the load for SOLR though, you may find that  
you want more than one server.

Do you mean that you're expecting about 1000 QPS over an index with up  
to 20 million documents?

--Matthew

On Nov 19, 2007, at 6:00 AM, Eswar K wrote:

> All,
>
> Can you give some information on this or atleast let me know where I  
> can
> find this information if its already listed out anywhere.
>
> Regards,
> Eswar
>
> On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:
>
>> Hi,
>>
>> I understand that Solr can be used on different Linux flavors. Is  
>> there
>> any preferred flavor (Like Red Hat, Ubuntu, etc)?
>> Also what is the kind of configuration of hardware (Processors,  
>> RAM, etc)
>> be best suited for the install?
>> We expect to load it with millions of documents (varying from 2 - 20
>> million). There might be around 1000 concurrent users.
>>
>> Your help in this regard will be appreciated.
>>
>> Regards,
>> Eswar
>>
>>

Re: Performance of Solr on different Platforms

[Eswar K <kj...@gmail.com>]

All,

Can you give some information on this or atleast let me know where I can
find this information if its already listed out anywhere.

Regards,
Eswar

On Nov 18, 2007 9:45 PM, Eswar K <kj...@gmail.com> wrote:

> Hi,
>
> I understand that Solr can be used on different Linux flavors. Is there
> any preferred flavor (Like Red Hat, Ubuntu, etc)?
> Also what is the kind of configuration of hardware (Processors, RAM, etc)
> be best suited for the install?
> We expect to load it with millions of documents (varying from 2 - 20
> million). There might be around 1000 concurrent users.
>
> Your help in this regard will be appreciated.
>
> Regards,
> Eswar
>
>