You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zhifeng Wang <zh...@gmail.com> on 2014/04/22 05:55:56 UTC
maximum number of shards per SolrCloud
Hi,
We are facing a high incoming rate of usually small documents (logs). The
incoming rate is initially assumed at 2K/sec but could reach as high as
20K/sec. So a year's worth of data could reach 60G (assuming the rate at
2K/sec) searchable documents.
Since a single shard can contain no more than 2G documents, we will need at
least 30 shards per year. Considering that we don't want to have shards to
their maximum capacity, the shards we need will be considerably higher.
My question is whether there is a hard (not possible) or soft (bad
performance) limit on the number of shards per SolrCloud. ZooKeeper
defaults file size to 1M, so I guess that causes some limit. If I set the
value to a larger number, will SolrCloud really scales OK if there
thousands of shards? Or I would be better off using multiple SolrCloud to
handle the data (Result aggregation is done outside of SolrCloud)?
Thanks,
Zhifeng
Re: maximum number of shards per SolrCloud
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Shawn
Thanks a lot. It is greatly helpful.
2014-04-23 0:43 GMT+08:00 Shawn Heisey <so...@elyograg.org>:
> On 4/22/2014 10:02 AM, yypvsxf19870706 wrote:
>
>> I am curious of the influences when have more than 2G docs in a
>> core.And we plan to have 5g docs/core.
>>
>> Please give me some suggestion about how to plan num of docs in a
>> core ?
>>
>
> One Solr core contains one Lucene index. It can't be divided further than
> that without a significant redesign. Quick note: Although SolrCloud can
> handle five billion documents with no problem, you can't have five billion
> documents in a single shard/core.
>
> The only hard limitation in the entire system is that you can't have more
> than approximately 2 billion documents in a single Lucene index. This is
> because a Java integer (which is a signed 32-bit number) is what gets used
> for internal Lucene document identifiers. Deleted documents count against
> that limit. It is theoretically possible to overcome this limitation, but
> it would be a MAJOR change to Lucene, requiring major changes in Solr as
> well.
>
> The other limitations you can run into with a large SolrCloud are mostly a
> matter of configuration, system resources, and scaling to multiple servers.
> They are not hard limitations in the software.
>
> I would never put more than about 1 billion documents in a single core.
> For performance reasons, it would be a good idea to never exceed a few
> hundred million. When a high query rate is required, loading only one Solr
> core per server may be a requirement.
>
> Thanks,
> Shawn
>
>
Re: maximum number of shards per SolrCloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/22/2014 10:02 AM, yypvsxf19870706 wrote:
> I am curious of the influences when have more than 2G docs in a core.And we plan to have 5g docs/core.
>
> Please give me some suggestion about how to plan num of docs in a core ?
One Solr core contains one Lucene index. It can't be divided further
than that without a significant redesign. Quick note: Although
SolrCloud can handle five billion documents with no problem, you can't
have five billion documents in a single shard/core.
The only hard limitation in the entire system is that you can't have
more than approximately 2 billion documents in a single Lucene index.
This is because a Java integer (which is a signed 32-bit number) is what
gets used for internal Lucene document identifiers. Deleted documents
count against that limit. It is theoretically possible to overcome this
limitation, but it would be a MAJOR change to Lucene, requiring major
changes in Solr as well.
The other limitations you can run into with a large SolrCloud are mostly
a matter of configuration, system resources, and scaling to multiple
servers. They are not hard limitations in the software.
I would never put more than about 1 billion documents in a single core.
For performance reasons, it would be a good idea to never exceed a few
hundred million. When a high query rate is required, loading only one
Solr core per server may be a requirement.
Thanks,
Shawn
Re: maximum number of shards per SolrCloud
Posted by yypvsxf19870706 <yy...@gmail.com>.
Hi
I am curious of the influences when have more than 2G docs in a core.And we plan to have 5g docs/core.
Please give me some suggestion about how to plan num of docs in a core ?
Thanks.
发自我的 iPhone
在 2014-4-22，12:30，Erick Erickson <er...@gmail.com> 写道：
> You're going to run into resource issues long before you hit 2G
> docs/node I suspect. I've seen from 50M t0 300M docs on a single node.
> Fortunately you may well be near the upper end of that since you're
> dealing with log files.
>
> Bottom line here is that you're off into largely uncharted territory
> when you start talking about hundreds of nodes. There's certainly work
> going on to make that work, but you'd be on the bleeding edge.
>
> Best,
> Erick
>
> On Mon, Apr 21, 2014 at 8:55 PM, Zhifeng Wang <zh...@gmail.com> wrote:
>> Hi,
>>
>> We are facing a high incoming rate of usually small documents (logs). The
>> incoming rate is initially assumed at 2K/sec but could reach as high as
>> 20K/sec. So a year's worth of data could reach 60G (assuming the rate at
>> 2K/sec) searchable documents.
>>
>> Since a single shard can contain no more than 2G documents, we will need at
>> least 30 shards per year. Considering that we don't want to have shards to
>> their maximum capacity, the shards we need will be considerably higher.
>>
>> My question is whether there is a hard (not possible) or soft (bad
>> performance) limit on the number of shards per SolrCloud. ZooKeeper
>> defaults file size to 1M, so I guess that causes some limit. If I set the
>> value to a larger number, will SolrCloud really scales OK if there
>> thousands of shards? Or I would be better off using multiple SolrCloud to
>> handle the data (Result aggregation is done outside of SolrCloud)?
>>
>> Thanks,
>> Zhifeng
Re: maximum number of shards per SolrCloud
Posted by Zhifeng Wang <zh...@gmail.com>.
Thanks, Erick.
I wonder whether having multiple SolrClouds helps in this matter. If I have
5 SolrClouds, that will effectively boost the number of shards 5 times. Of
course this introduces issues when I need to aggregate results from
different SolrClouds.
Thanks,
Zhifeng
On Tue, Apr 22, 2014 at 12:30 AM, Erick Erickson <er...@gmail.com>wrote:
> You're going to run into resource issues long before you hit 2G
> docs/node I suspect. I've seen from 50M t0 300M docs on a single node.
> Fortunately you may well be near the upper end of that since you're
> dealing with log files.
>
> Bottom line here is that you're off into largely uncharted territory
> when you start talking about hundreds of nodes. There's certainly work
> going on to make that work, but you'd be on the bleeding edge.
>
> Best,
> Erick
>
> On Mon, Apr 21, 2014 at 8:55 PM, Zhifeng Wang <zh...@gmail.com>
> wrote:
> > Hi,
> >
> > We are facing a high incoming rate of usually small documents (logs). The
> > incoming rate is initially assumed at 2K/sec but could reach as high as
> > 20K/sec. So a year's worth of data could reach 60G (assuming the rate at
> > 2K/sec) searchable documents.
> >
> > Since a single shard can contain no more than 2G documents, we will need
> at
> > least 30 shards per year. Considering that we don't want to have shards
> to
> > their maximum capacity, the shards we need will be considerably higher.
> >
> > My question is whether there is a hard (not possible) or soft (bad
> > performance) limit on the number of shards per SolrCloud. ZooKeeper
> > defaults file size to 1M, so I guess that causes some limit. If I set the
> > value to a larger number, will SolrCloud really scales OK if there
> > thousands of shards? Or I would be better off using multiple SolrCloud
> to
> > handle the data (Result aggregation is done outside of SolrCloud)?
> >
> > Thanks,
> > Zhifeng
>
Re: maximum number of shards per SolrCloud
Posted by Erick Erickson <er...@gmail.com>.
You're going to run into resource issues long before you hit 2G
docs/node I suspect. I've seen from 50M t0 300M docs on a single node.
Fortunately you may well be near the upper end of that since you're
dealing with log files.
Bottom line here is that you're off into largely uncharted territory
when you start talking about hundreds of nodes. There's certainly work
going on to make that work, but you'd be on the bleeding edge.
Best,
Erick
On Mon, Apr 21, 2014 at 8:55 PM, Zhifeng Wang <zh...@gmail.com> wrote:
> Hi,
>
> We are facing a high incoming rate of usually small documents (logs). The
> incoming rate is initially assumed at 2K/sec but could reach as high as
> 20K/sec. So a year's worth of data could reach 60G (assuming the rate at
> 2K/sec) searchable documents.
>
> Since a single shard can contain no more than 2G documents, we will need at
> least 30 shards per year. Considering that we don't want to have shards to
> their maximum capacity, the shards we need will be considerably higher.
>
> My question is whether there is a hard (not possible) or soft (bad
> performance) limit on the number of shards per SolrCloud. ZooKeeper
> defaults file size to 1M, so I guess that causes some limit. If I set the
> value to a larger number, will SolrCloud really scales OK if there
> thousands of shards? Or I would be better off using multiple SolrCloud to
> handle the data (Result aggregation is done outside of SolrCloud)?
>
> Thanks,
> Zhifeng