You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Priti Solanki <pr...@gmail.com> on 2014/03/06 17:51:28 UTC
SolrCloud setup guidance
Hello Everyone,
I would like to take you guidance of following
I have a single core with 124 GB of index data size. Indexing and Reading
both are very slow as I have 7 GB RAM to support this huge data. Almost 8
million of documents.
Hence, we thought of going to SolrCloud so that we can accommodate more
upcoming data. I have data for 13 country with their millions of products
and we want to set up solrcloud for the same.
I am in need of some initial thoughts about how to setup solrCloud for such
requirement. How to we come to know how many nodes,core I would be needing
to support this...
we are thinking to host this on Amazon...Any guidance or reading links
,case study will be highly appreciated.
Regards,
Re: SolrCloud setup guidance
Posted by Priti Solanki <pr...@gmail.com>.
Thanks Furkam,
This give some really good understanding. We have Amazon Instance and right
now it is running on m1.large.
In Amazon we are not finding a support to increase ONLY RAM ! that our main
concern and we are actively looking which instance can help us to support
this index size.
Do you think m2.4xlarge would help?.
Also if you can share some input over speeding up the indexer process?.
Regards,
Priti
On Mon, Mar 10, 2014 at 6:30 PM, Furkan KAMACI <fu...@gmail.com>wrote:
> Hi Priti;
>
> 100 qps is not much but 7 GB is too low and it may be a problem for you. I
> have a tens of nodes of SolrCloud and I send data them via Map/Reduce via
> tens of servers. However indexing speed did not be a problem for me yet.
> Problems occurs because of network communication, RAM or something else.
>
> If we talk about the ideal size of RAM: then you should have a RAM total of
> at least your index size. Because you may want to cache everything ideally.
> However if you consider the situations as like optimize you it would be
> nice 3 times larger of your index size. However RAM is also depends on your
> user characteristics. Sometimes caching only some of documents may be
> enough because of they are queried many many times.
>
> As I mentioned previously 100 qps is not much and indexing is not a
> bottleneck at many situations I suggest to remember the occam's razor.
> Start with simple and increase your RAM step by step. On the other hand I
> suggest you to use a tool as like Solrmeter to test your qps.
>
> If you have any question I can help you about your infrastructure.
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-03-10 12:41 GMT+02:00 Priti Solanki <pr...@gmail.com>:
>
> > As of now index is on 136 GB.
> >
> > I want to understand can we do multiple write on solr? I don't have any
> > partitioning strategy as of now.
> >
> > On Amazon instance for Solr the disk read/write a like 5% or so . I am
> not
> > able to understand even though I am almost processing 300 records per min
> > how come Solr server is not receiving high disk IO.
> >
> > I have a query, When solr is writing does it update complete index again
> ?
> >
> > I will try and increase the RAM to 128 and see if some improvement
> comes..
> >
> >
> > On Mon, Mar 10, 2014 at 9:21 AM, Susheel Kumar <
> > susheel.kumar@thedigitalgroup.net> wrote:
> >
> > > Not sure how fast your index will grow but first you may still want to
> > > consider upgrading the single machine to 128 GB to see how the
> > performance
> > > is coming. Current memory 7 GB is really low. After that you may want
> to
> > > add another node to partition the index into 2 nodes/shards (assuming
> you
> > > have some partition strategy ) that index size of 150-200 GB can fit
> into
> > > two nodes memory.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > -----Original Message-----
> > > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > > Sent: Friday, March 07, 2014 12:50 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: SolrCloud setup guidance
> > >
> > > Thanks Susheel,
> > >
> > > But this index will keep on growing that my worry So I always have to
> > > increase the RAM .
> > >
> > > Can you suggest how many nodes one can think to support this bug index?
> > >
> > > Regards,
> > >
> > >
> > >
> > > On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
> > > susheel.kumar@thedigitalgroup.net> wrote:
> > >
> > > > Setting up Solr cloud(horizontal scaling) is definitely a good idea
> > > > for this big index but before going to Solr cloud, are you able to
> > > > upgrade your single node to 128GB of memory(vertical scaling) to see
> > the
> > > difference.
> > > >
> > > > Thanks,
> > > > Susheel
> > > >
> > > > -----Original Message-----
> > > > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > > > Sent: Thursday, March 06, 2014 10:51 AM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: SolrCloud setup guidance
> > > >
> > > > Hello Everyone,
> > > >
> > > > I would like to take you guidance of following
> > > >
> > > > I have a single core with 124 GB of index data size. Indexing and
> > > > Reading both are very slow as I have 7 GB RAM to support this huge
> > > > data. Almost 8 million of documents.
> > > >
> > > > Hence, we thought of going to SolrCloud so that we can accommodate
> > > > more upcoming data. I have data for 13 country with their millions of
> > > > products and we want to set up solrcloud for the same.
> > > >
> > > > I am in need of some initial thoughts about how to setup solrCloud
> for
> > > > such requirement. How to we come to know how many nodes,core I would
> > > > be needing to support this...
> > > >
> > > > we are thinking to host this on Amazon...Any guidance or reading
> links
> > > > ,case study will be highly appreciated.
> > > >
> > > > Regards,
> > > >
> > >
> >
>
Re: SolrCloud setup guidance
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Priti;
100 qps is not much but 7 GB is too low and it may be a problem for you. I
have a tens of nodes of SolrCloud and I send data them via Map/Reduce via
tens of servers. However indexing speed did not be a problem for me yet.
Problems occurs because of network communication, RAM or something else.
If we talk about the ideal size of RAM: then you should have a RAM total of
at least your index size. Because you may want to cache everything ideally.
However if you consider the situations as like optimize you it would be
nice 3 times larger of your index size. However RAM is also depends on your
user characteristics. Sometimes caching only some of documents may be
enough because of they are queried many many times.
As I mentioned previously 100 qps is not much and indexing is not a
bottleneck at many situations I suggest to remember the occam's razor.
Start with simple and increase your RAM step by step. On the other hand I
suggest you to use a tool as like Solrmeter to test your qps.
If you have any question I can help you about your infrastructure.
Thanks;
Furkan KAMACI
2014-03-10 12:41 GMT+02:00 Priti Solanki <pr...@gmail.com>:
> As of now index is on 136 GB.
>
> I want to understand can we do multiple write on solr? I don't have any
> partitioning strategy as of now.
>
> On Amazon instance for Solr the disk read/write a like 5% or so . I am not
> able to understand even though I am almost processing 300 records per min
> how come Solr server is not receiving high disk IO.
>
> I have a query, When solr is writing does it update complete index again ?
>
> I will try and increase the RAM to 128 and see if some improvement comes..
>
>
> On Mon, Mar 10, 2014 at 9:21 AM, Susheel Kumar <
> susheel.kumar@thedigitalgroup.net> wrote:
>
> > Not sure how fast your index will grow but first you may still want to
> > consider upgrading the single machine to 128 GB to see how the
> performance
> > is coming. Current memory 7 GB is really low. After that you may want to
> > add another node to partition the index into 2 nodes/shards (assuming you
> > have some partition strategy ) that index size of 150-200 GB can fit into
> > two nodes memory.
> >
> > Thanks,
> > Susheel
> >
> > -----Original Message-----
> > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > Sent: Friday, March 07, 2014 12:50 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: SolrCloud setup guidance
> >
> > Thanks Susheel,
> >
> > But this index will keep on growing that my worry So I always have to
> > increase the RAM .
> >
> > Can you suggest how many nodes one can think to support this bug index?
> >
> > Regards,
> >
> >
> >
> > On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
> > susheel.kumar@thedigitalgroup.net> wrote:
> >
> > > Setting up Solr cloud(horizontal scaling) is definitely a good idea
> > > for this big index but before going to Solr cloud, are you able to
> > > upgrade your single node to 128GB of memory(vertical scaling) to see
> the
> > difference.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > -----Original Message-----
> > > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > > Sent: Thursday, March 06, 2014 10:51 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: SolrCloud setup guidance
> > >
> > > Hello Everyone,
> > >
> > > I would like to take you guidance of following
> > >
> > > I have a single core with 124 GB of index data size. Indexing and
> > > Reading both are very slow as I have 7 GB RAM to support this huge
> > > data. Almost 8 million of documents.
> > >
> > > Hence, we thought of going to SolrCloud so that we can accommodate
> > > more upcoming data. I have data for 13 country with their millions of
> > > products and we want to set up solrcloud for the same.
> > >
> > > I am in need of some initial thoughts about how to setup solrCloud for
> > > such requirement. How to we come to know how many nodes,core I would
> > > be needing to support this...
> > >
> > > we are thinking to host this on Amazon...Any guidance or reading links
> > > ,case study will be highly appreciated.
> > >
> > > Regards,
> > >
> >
>
Re: SolrCloud setup guidance
Posted by Priti Solanki <pr...@gmail.com>.
As of now index is on 136 GB.
I want to understand can we do multiple write on solr? I don't have any
partitioning strategy as of now.
On Amazon instance for Solr the disk read/write a like 5% or so . I am not
able to understand even though I am almost processing 300 records per min
how come Solr server is not receiving high disk IO.
I have a query, When solr is writing does it update complete index again ?
I will try and increase the RAM to 128 and see if some improvement comes..
On Mon, Mar 10, 2014 at 9:21 AM, Susheel Kumar <
susheel.kumar@thedigitalgroup.net> wrote:
> Not sure how fast your index will grow but first you may still want to
> consider upgrading the single machine to 128 GB to see how the performance
> is coming. Current memory 7 GB is really low. After that you may want to
> add another node to partition the index into 2 nodes/shards (assuming you
> have some partition strategy ) that index size of 150-200 GB can fit into
> two nodes memory.
>
> Thanks,
> Susheel
>
> -----Original Message-----
> From: Priti Solanki [mailto:pritiatwork@gmail.com]
> Sent: Friday, March 07, 2014 12:50 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud setup guidance
>
> Thanks Susheel,
>
> But this index will keep on growing that my worry So I always have to
> increase the RAM .
>
> Can you suggest how many nodes one can think to support this bug index?
>
> Regards,
>
>
>
> On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
> susheel.kumar@thedigitalgroup.net> wrote:
>
> > Setting up Solr cloud(horizontal scaling) is definitely a good idea
> > for this big index but before going to Solr cloud, are you able to
> > upgrade your single node to 128GB of memory(vertical scaling) to see the
> difference.
> >
> > Thanks,
> > Susheel
> >
> > -----Original Message-----
> > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > Sent: Thursday, March 06, 2014 10:51 AM
> > To: solr-user@lucene.apache.org
> > Subject: SolrCloud setup guidance
> >
> > Hello Everyone,
> >
> > I would like to take you guidance of following
> >
> > I have a single core with 124 GB of index data size. Indexing and
> > Reading both are very slow as I have 7 GB RAM to support this huge
> > data. Almost 8 million of documents.
> >
> > Hence, we thought of going to SolrCloud so that we can accommodate
> > more upcoming data. I have data for 13 country with their millions of
> > products and we want to set up solrcloud for the same.
> >
> > I am in need of some initial thoughts about how to setup solrCloud for
> > such requirement. How to we come to know how many nodes,core I would
> > be needing to support this...
> >
> > we are thinking to host this on Amazon...Any guidance or reading links
> > ,case study will be highly appreciated.
> >
> > Regards,
> >
>
RE: SolrCloud setup guidance
Posted by Susheel Kumar <su...@thedigitalgroup.net>.
Not sure how fast your index will grow but first you may still want to consider upgrading the single machine to 128 GB to see how the performance is coming. Current memory 7 GB is really low. After that you may want to add another node to partition the index into 2 nodes/shards (assuming you have some partition strategy ) that index size of 150-200 GB can fit into two nodes memory.
Thanks,
Susheel
-----Original Message-----
From: Priti Solanki [mailto:pritiatwork@gmail.com]
Sent: Friday, March 07, 2014 12:50 AM
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud setup guidance
Thanks Susheel,
But this index will keep on growing that my worry So I always have to increase the RAM .
Can you suggest how many nodes one can think to support this bug index?
Regards,
On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar < susheel.kumar@thedigitalgroup.net> wrote:
> Setting up Solr cloud(horizontal scaling) is definitely a good idea
> for this big index but before going to Solr cloud, are you able to
> upgrade your single node to 128GB of memory(vertical scaling) to see the difference.
>
> Thanks,
> Susheel
>
> -----Original Message-----
> From: Priti Solanki [mailto:pritiatwork@gmail.com]
> Sent: Thursday, March 06, 2014 10:51 AM
> To: solr-user@lucene.apache.org
> Subject: SolrCloud setup guidance
>
> Hello Everyone,
>
> I would like to take you guidance of following
>
> I have a single core with 124 GB of index data size. Indexing and
> Reading both are very slow as I have 7 GB RAM to support this huge
> data. Almost 8 million of documents.
>
> Hence, we thought of going to SolrCloud so that we can accommodate
> more upcoming data. I have data for 13 country with their millions of
> products and we want to set up solrcloud for the same.
>
> I am in need of some initial thoughts about how to setup solrCloud for
> such requirement. How to we come to know how many nodes,core I would
> be needing to support this...
>
> we are thinking to host this on Amazon...Any guidance or reading links
> ,case study will be highly appreciated.
>
> Regards,
>
Re: SolrCloud setup guidance
Posted by Priti Solanki <pr...@gmail.com>.
Furkan, 100 request second would be ideal in out situation.
Regards,
Priti
On Sat, Mar 8, 2014 at 3:41 AM, Furkan KAMACI <fu...@gmail.com>wrote:
> Hi;
>
> What's your performance expectation for qps (query per second)?
>
> Thanks;
> Furkan KAMACI
> 7 Mar 2014 08:50 tarihinde "Priti Solanki" <pr...@gmail.com> yazdı:
>
> > Thanks Susheel,
> >
> > But this index will keep on growing that my worry So I always have to
> > increase the RAM .
> >
> > Can you suggest how many nodes one can think to support this bug index?
> >
> > Regards,
> >
> >
> >
> > On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
> > susheel.kumar@thedigitalgroup.net> wrote:
> >
> > > Setting up Solr cloud(horizontal scaling) is definitely a good idea for
> > > this big index but before going to Solr cloud, are you able to upgrade
> > your
> > > single node to 128GB of memory(vertical scaling) to see the difference.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > -----Original Message-----
> > > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > > Sent: Thursday, March 06, 2014 10:51 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: SolrCloud setup guidance
> > >
> > > Hello Everyone,
> > >
> > > I would like to take you guidance of following
> > >
> > > I have a single core with 124 GB of index data size. Indexing and
> Reading
> > > both are very slow as I have 7 GB RAM to support this huge data.
> Almost
> > 8
> > > million of documents.
> > >
> > > Hence, we thought of going to SolrCloud so that we can accommodate more
> > > upcoming data. I have data for 13 country with their millions of
> products
> > > and we want to set up solrcloud for the same.
> > >
> > > I am in need of some initial thoughts about how to setup solrCloud for
> > > such requirement. How to we come to know how many nodes,core I would be
> > > needing to support this...
> > >
> > > we are thinking to host this on Amazon...Any guidance or reading links
> > > ,case study will be highly appreciated.
> > >
> > > Regards,
> > >
> >
>
Re: SolrCloud setup guidance
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
What's your performance expectation for qps (query per second)?
Thanks;
Furkan KAMACI
7 Mar 2014 08:50 tarihinde "Priti Solanki" <pr...@gmail.com> yazdı:
> Thanks Susheel,
>
> But this index will keep on growing that my worry So I always have to
> increase the RAM .
>
> Can you suggest how many nodes one can think to support this bug index?
>
> Regards,
>
>
>
> On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
> susheel.kumar@thedigitalgroup.net> wrote:
>
> > Setting up Solr cloud(horizontal scaling) is definitely a good idea for
> > this big index but before going to Solr cloud, are you able to upgrade
> your
> > single node to 128GB of memory(vertical scaling) to see the difference.
> >
> > Thanks,
> > Susheel
> >
> > -----Original Message-----
> > From: Priti Solanki [mailto:pritiatwork@gmail.com]
> > Sent: Thursday, March 06, 2014 10:51 AM
> > To: solr-user@lucene.apache.org
> > Subject: SolrCloud setup guidance
> >
> > Hello Everyone,
> >
> > I would like to take you guidance of following
> >
> > I have a single core with 124 GB of index data size. Indexing and Reading
> > both are very slow as I have 7 GB RAM to support this huge data. Almost
> 8
> > million of documents.
> >
> > Hence, we thought of going to SolrCloud so that we can accommodate more
> > upcoming data. I have data for 13 country with their millions of products
> > and we want to set up solrcloud for the same.
> >
> > I am in need of some initial thoughts about how to setup solrCloud for
> > such requirement. How to we come to know how many nodes,core I would be
> > needing to support this...
> >
> > we are thinking to host this on Amazon...Any guidance or reading links
> > ,case study will be highly appreciated.
> >
> > Regards,
> >
>
Re: SolrCloud setup guidance
Posted by Priti Solanki <pr...@gmail.com>.
Thanks Susheel,
But this index will keep on growing that my worry So I always have to
increase the RAM .
Can you suggest how many nodes one can think to support this bug index?
Regards,
On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar <
susheel.kumar@thedigitalgroup.net> wrote:
> Setting up Solr cloud(horizontal scaling) is definitely a good idea for
> this big index but before going to Solr cloud, are you able to upgrade your
> single node to 128GB of memory(vertical scaling) to see the difference.
>
> Thanks,
> Susheel
>
> -----Original Message-----
> From: Priti Solanki [mailto:pritiatwork@gmail.com]
> Sent: Thursday, March 06, 2014 10:51 AM
> To: solr-user@lucene.apache.org
> Subject: SolrCloud setup guidance
>
> Hello Everyone,
>
> I would like to take you guidance of following
>
> I have a single core with 124 GB of index data size. Indexing and Reading
> both are very slow as I have 7 GB RAM to support this huge data. Almost 8
> million of documents.
>
> Hence, we thought of going to SolrCloud so that we can accommodate more
> upcoming data. I have data for 13 country with their millions of products
> and we want to set up solrcloud for the same.
>
> I am in need of some initial thoughts about how to setup solrCloud for
> such requirement. How to we come to know how many nodes,core I would be
> needing to support this...
>
> we are thinking to host this on Amazon...Any guidance or reading links
> ,case study will be highly appreciated.
>
> Regards,
>
RE: SolrCloud setup guidance
Posted by Susheel Kumar <su...@thedigitalgroup.net>.
Setting up Solr cloud(horizontal scaling) is definitely a good idea for this big index but before going to Solr cloud, are you able to upgrade your single node to 128GB of memory(vertical scaling) to see the difference.
Thanks,
Susheel
-----Original Message-----
From: Priti Solanki [mailto:pritiatwork@gmail.com]
Sent: Thursday, March 06, 2014 10:51 AM
To: solr-user@lucene.apache.org
Subject: SolrCloud setup guidance
Hello Everyone,
I would like to take you guidance of following
I have a single core with 124 GB of index data size. Indexing and Reading both are very slow as I have 7 GB RAM to support this huge data. Almost 8 million of documents.
Hence, we thought of going to SolrCloud so that we can accommodate more upcoming data. I have data for 13 country with their millions of products and we want to set up solrcloud for the same.
I am in need of some initial thoughts about how to setup solrCloud for such requirement. How to we come to know how many nodes,core I would be needing to support this...
we are thinking to host this on Amazon...Any guidance or reading links ,case study will be highly appreciated.
Regards,