You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 苗海泉 <ms...@gmail.com> on 2018/03/10 05:52:00 UTC
The Impact of the Number of Collections on Indexing Performance in
Solr 6.0
hello,We found a problem. In solr 6.0, the indexing speed of solr is
influenced by the number of solr collections. The speed is normal before
the limit is reached. If the limit is reached, the indexing speed will
decrease by 50 times.
In our environment, there are 49 solr nodes. If each collection has 25
shards, you can maintain high-speed indexing until the total number of
collections is about 900. To reduce the number of collections to the limit,
the speed will increase. Go up.
If each collection is 49 shards, the total number of collections can only
be about 700, exceeding this value will cause the index to drop
dramatically.
In the explanation, we are single copies, and multiple copies will cause
serious stability problems in the large solr cluster environment.
At first I suspect that it was due to too many thread submissions, and
there are still problems with this method, so I'm inclined to
searcherExecutor thread pool thread. This is just my guess, I want to know
the real reason. Can someone know if I can help?
Also, I noticed that the searcherExecutor thread and solr collection's
shards basically correspond to each other. How can I reduce the number of
threads or even close it? Although there are many collections in our
environment, there are few queries and it is not necessary to keep the
threads open to provide queries. This is too wasteful.
thank you .
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by spoonerk <jo...@gmail.com>.
I have tried emailing to.unsubscribe. I have tried disrupting threads
hoping to anger the admin into getting me out of the spam list. All I get
is arrogant emails about headers
On Mar 12, 2018 1:15 AM, "苗海泉" <ms...@gmail.com> wrote:
> Thanks Erick and Shawn , Thank you for your patience. I said that the
> above phenomenon was caused by the IO, cpu, memory, and network io. The
> swap was turned off and the machine's memory was sufficient. When the speed
> of indexing is declining, QTime is found to take 3 seconds to 4 seconds to
> reload the index, so it can be guessed that it is more likely to be a Solr
> problem than a jetty. It is worth mentioning that when the speed of the
> index under construction dropped sharply, the Solr used only about 5% of
> the CPU, and when it was normal, the CPU usage was 200 percent, and the
> overall system's CPU usage was 100 percent. About twenty.
>
> Basic information:
> 1) The data volume of each collection is between 2 billion and 3 billion.
> 2) The configuration of the machine is 24 cpu and 128G memory.
> 3) The disk usage per copy is about 10G.
>
> In addition, I noticed that the work of zookeeper is normal and there is no
> error or warning message.
>
> So all these phenomena make me think that the internal specific mechanism
> of solr may lead to a sharp drop in the index construction speed. At
> present, it seems that our solr's machine resources are sufficient.
>
> As for the reduction of the number of collections that you said, we also
> have this plan, and we are looking for ways to reform it. Are there any
> other suggestions?
>
>
> Best .
> miaohq
>
> 2018-03-11 10:15 GMT+08:00 spoonerk <jo...@gmail.com>:
>
> > Wow thanks. Just trying to unsubscribe. Most email lists let u do that
> >
> > On Mar 10, 2018 2:36 PM, "Erick Erickson" <er...@gmail.com>
> wrote:
> >
> > > Spoonerk:
> > >
> > > You say you've tried "many times", but you haven't provided full
> > > header as described in the "problems" link at the link below. You
> > > haven't e-mailed the list owner as suggested in the "problems" link.
> > > You haven't, in short, provided any of the information that's
> > > necessary to actually unsubscribe you.
> > >
> > > Please follow the instructions here:
> > > http://lucene.apache.org/solr/community.html#mailing-lists-irc. In
> > > particular look at the "problems" link.
> > >
> > > You must use the _exact_ same e-mail as you used to subscribe.
> > >
> > > If the initial try doesn't work and following the suggestions at the
> > > "problems" link doesn't work for you, let us know. But note you need
> > > to show us the _entire_ return header to allow anyone to diagnose the
> > > problem.
> > >
> > > Best,
> > > Erick
> > >
> > > On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <jo...@gmail.com>
> > wrote:
> > > > I have manually unsubscribed many times. But I still get emails from
> > the
> > > > list. Can some admin please unsubscribe me?
> > > >
> > > > On Mar 9, 2018 9:52 PM, "苗海泉" <ms...@gmail.com> wrote:
> > > >
> > > >> hello,We found a problem. In solr 6.0, the indexing speed of solr is
> > > >> influenced by the number of solr collections. The speed is normal
> > before
> > > >> the limit is reached. If the limit is reached, the indexing speed
> will
> > > >> decrease by 50 times.
> > > >>
> > > >> In our environment, there are 49 solr nodes. If each collection has
> 25
> > > >> shards, you can maintain high-speed indexing until the total number
> of
> > > >> collections is about 900. To reduce the number of collections to the
> > > limit,
> > > >> the speed will increase. Go up.
> > > >> If each collection is 49 shards, the total number of collections can
> > > only
> > > >> be about 700, exceeding this value will cause the index to drop
> > > >> dramatically.
> > > >> In the explanation, we are single copies, and multiple copies will
> > cause
> > > >> serious stability problems in the large solr cluster environment.
> > > >>
> > > >> At first I suspect that it was due to too many thread submissions,
> and
> > > >> there are still problems with this method, so I'm inclined to
> > > >> searcherExecutor thread pool thread. This is just my guess, I want
> to
> > > know
> > > >> the real reason. Can someone know if I can help?
> > > >>
> > > >> Also, I noticed that the searcherExecutor thread and solr
> collection's
> > > >> shards basically correspond to each other. How can I reduce the
> number
> > > of
> > > >> threads or even close it? Although there are many collections in our
> > > >> environment, there are few queries and it is not necessary to keep
> the
> > > >> threads open to provide queries. This is too wasteful.
> > > >>
> > > >> thank you .
> > > >>
> > >
> >
>
>
>
> --
> ==============================
> 联创科技
> 知行如一
> ==============================
>
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by 苗海泉 <ms...@gmail.com>.
Thanks Erick and Shawn , Thank you for your patience. I said that the
above phenomenon was caused by the IO, cpu, memory, and network io. The
swap was turned off and the machine's memory was sufficient. When the speed
of indexing is declining, QTime is found to take 3 seconds to 4 seconds to
reload the index, so it can be guessed that it is more likely to be a Solr
problem than a jetty. It is worth mentioning that when the speed of the
index under construction dropped sharply, the Solr used only about 5% of
the CPU, and when it was normal, the CPU usage was 200 percent, and the
overall system's CPU usage was 100 percent. About twenty.
Basic information:
1) The data volume of each collection is between 2 billion and 3 billion.
2) The configuration of the machine is 24 cpu and 128G memory.
3) The disk usage per copy is about 10G.
In addition, I noticed that the work of zookeeper is normal and there is no
error or warning message.
So all these phenomena make me think that the internal specific mechanism
of solr may lead to a sharp drop in the index construction speed. At
present, it seems that our solr's machine resources are sufficient.
As for the reduction of the number of collections that you said, we also
have this plan, and we are looking for ways to reform it. Are there any
other suggestions?
Best .
miaohq
2018-03-11 10:15 GMT+08:00 spoonerk <jo...@gmail.com>:
> Wow thanks. Just trying to unsubscribe. Most email lists let u do that
>
> On Mar 10, 2018 2:36 PM, "Erick Erickson" <er...@gmail.com> wrote:
>
> > Spoonerk:
> >
> > You say you've tried "many times", but you haven't provided full
> > header as described in the "problems" link at the link below. You
> > haven't e-mailed the list owner as suggested in the "problems" link.
> > You haven't, in short, provided any of the information that's
> > necessary to actually unsubscribe you.
> >
> > Please follow the instructions here:
> > http://lucene.apache.org/solr/community.html#mailing-lists-irc. In
> > particular look at the "problems" link.
> >
> > You must use the _exact_ same e-mail as you used to subscribe.
> >
> > If the initial try doesn't work and following the suggestions at the
> > "problems" link doesn't work for you, let us know. But note you need
> > to show us the _entire_ return header to allow anyone to diagnose the
> > problem.
> >
> > Best,
> > Erick
> >
> > On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <jo...@gmail.com>
> wrote:
> > > I have manually unsubscribed many times. But I still get emails from
> the
> > > list. Can some admin please unsubscribe me?
> > >
> > > On Mar 9, 2018 9:52 PM, "苗海泉" <ms...@gmail.com> wrote:
> > >
> > >> hello,We found a problem. In solr 6.0, the indexing speed of solr is
> > >> influenced by the number of solr collections. The speed is normal
> before
> > >> the limit is reached. If the limit is reached, the indexing speed will
> > >> decrease by 50 times.
> > >>
> > >> In our environment, there are 49 solr nodes. If each collection has 25
> > >> shards, you can maintain high-speed indexing until the total number of
> > >> collections is about 900. To reduce the number of collections to the
> > limit,
> > >> the speed will increase. Go up.
> > >> If each collection is 49 shards, the total number of collections can
> > only
> > >> be about 700, exceeding this value will cause the index to drop
> > >> dramatically.
> > >> In the explanation, we are single copies, and multiple copies will
> cause
> > >> serious stability problems in the large solr cluster environment.
> > >>
> > >> At first I suspect that it was due to too many thread submissions, and
> > >> there are still problems with this method, so I'm inclined to
> > >> searcherExecutor thread pool thread. This is just my guess, I want to
> > know
> > >> the real reason. Can someone know if I can help?
> > >>
> > >> Also, I noticed that the searcherExecutor thread and solr collection's
> > >> shards basically correspond to each other. How can I reduce the number
> > of
> > >> threads or even close it? Although there are many collections in our
> > >> environment, there are few queries and it is not necessary to keep the
> > >> threads open to provide queries. This is too wasteful.
> > >>
> > >> thank you .
> > >>
> >
>
--
==============================
联创科技
知行如一
==============================
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by spoonerk <jo...@gmail.com>.
Wow thanks. Just trying to unsubscribe. Most email lists let u do that
On Mar 10, 2018 2:36 PM, "Erick Erickson" <er...@gmail.com> wrote:
> Spoonerk:
>
> You say you've tried "many times", but you haven't provided full
> header as described in the "problems" link at the link below. You
> haven't e-mailed the list owner as suggested in the "problems" link.
> You haven't, in short, provided any of the information that's
> necessary to actually unsubscribe you.
>
> Please follow the instructions here:
> http://lucene.apache.org/solr/community.html#mailing-lists-irc. In
> particular look at the "problems" link.
>
> You must use the _exact_ same e-mail as you used to subscribe.
>
> If the initial try doesn't work and following the suggestions at the
> "problems" link doesn't work for you, let us know. But note you need
> to show us the _entire_ return header to allow anyone to diagnose the
> problem.
>
> Best,
> Erick
>
> On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <jo...@gmail.com> wrote:
> > I have manually unsubscribed many times. But I still get emails from the
> > list. Can some admin please unsubscribe me?
> >
> > On Mar 9, 2018 9:52 PM, "苗海泉" <ms...@gmail.com> wrote:
> >
> >> hello,We found a problem. In solr 6.0, the indexing speed of solr is
> >> influenced by the number of solr collections. The speed is normal before
> >> the limit is reached. If the limit is reached, the indexing speed will
> >> decrease by 50 times.
> >>
> >> In our environment, there are 49 solr nodes. If each collection has 25
> >> shards, you can maintain high-speed indexing until the total number of
> >> collections is about 900. To reduce the number of collections to the
> limit,
> >> the speed will increase. Go up.
> >> If each collection is 49 shards, the total number of collections can
> only
> >> be about 700, exceeding this value will cause the index to drop
> >> dramatically.
> >> In the explanation, we are single copies, and multiple copies will cause
> >> serious stability problems in the large solr cluster environment.
> >>
> >> At first I suspect that it was due to too many thread submissions, and
> >> there are still problems with this method, so I'm inclined to
> >> searcherExecutor thread pool thread. This is just my guess, I want to
> know
> >> the real reason. Can someone know if I can help?
> >>
> >> Also, I noticed that the searcherExecutor thread and solr collection's
> >> shards basically correspond to each other. How can I reduce the number
> of
> >> threads or even close it? Although there are many collections in our
> >> environment, there are few queries and it is not necessary to keep the
> >> threads open to provide queries. This is too wasteful.
> >>
> >> thank you .
> >>
>
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by Erick Erickson <er...@gmail.com>.
Spoonerk:
You say you've tried "many times", but you haven't provided full
header as described in the "problems" link at the link below. You
haven't e-mailed the list owner as suggested in the "problems" link.
You haven't, in short, provided any of the information that's
necessary to actually unsubscribe you.
Please follow the instructions here:
http://lucene.apache.org/solr/community.html#mailing-lists-irc. In
particular look at the "problems" link.
You must use the _exact_ same e-mail as you used to subscribe.
If the initial try doesn't work and following the suggestions at the
"problems" link doesn't work for you, let us know. But note you need
to show us the _entire_ return header to allow anyone to diagnose the
problem.
Best,
Erick
On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <jo...@gmail.com> wrote:
> I have manually unsubscribed many times. But I still get emails from the
> list. Can some admin please unsubscribe me?
>
> On Mar 9, 2018 9:52 PM, "苗海泉" <ms...@gmail.com> wrote:
>
>> hello,We found a problem. In solr 6.0, the indexing speed of solr is
>> influenced by the number of solr collections. The speed is normal before
>> the limit is reached. If the limit is reached, the indexing speed will
>> decrease by 50 times.
>>
>> In our environment, there are 49 solr nodes. If each collection has 25
>> shards, you can maintain high-speed indexing until the total number of
>> collections is about 900. To reduce the number of collections to the limit,
>> the speed will increase. Go up.
>> If each collection is 49 shards, the total number of collections can only
>> be about 700, exceeding this value will cause the index to drop
>> dramatically.
>> In the explanation, we are single copies, and multiple copies will cause
>> serious stability problems in the large solr cluster environment.
>>
>> At first I suspect that it was due to too many thread submissions, and
>> there are still problems with this method, so I'm inclined to
>> searcherExecutor thread pool thread. This is just my guess, I want to know
>> the real reason. Can someone know if I can help?
>>
>> Also, I noticed that the searcherExecutor thread and solr collection's
>> shards basically correspond to each other. How can I reduce the number of
>> threads or even close it? Although there are many collections in our
>> environment, there are few queries and it is not necessary to keep the
>> threads open to provide queries. This is too wasteful.
>>
>> thank you .
>>
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by spoonerk <jo...@gmail.com>.
I have manually unsubscribed many times. But I still get emails from the
list. Can some admin please unsubscribe me?
On Mar 9, 2018 9:52 PM, "苗海泉" <ms...@gmail.com> wrote:
> hello,We found a problem. In solr 6.0, the indexing speed of solr is
> influenced by the number of solr collections. The speed is normal before
> the limit is reached. If the limit is reached, the indexing speed will
> decrease by 50 times.
>
> In our environment, there are 49 solr nodes. If each collection has 25
> shards, you can maintain high-speed indexing until the total number of
> collections is about 900. To reduce the number of collections to the limit,
> the speed will increase. Go up.
> If each collection is 49 shards, the total number of collections can only
> be about 700, exceeding this value will cause the index to drop
> dramatically.
> In the explanation, we are single copies, and multiple copies will cause
> serious stability problems in the large solr cluster environment.
>
> At first I suspect that it was due to too many thread submissions, and
> there are still problems with this method, so I'm inclined to
> searcherExecutor thread pool thread. This is just my guess, I want to know
> the real reason. Can someone know if I can help?
>
> Also, I noticed that the searcherExecutor thread and solr collection's
> shards basically correspond to each other. How can I reduce the number of
> threads or even close it? Although there are many collections in our
> environment, there are few queries and it is not necessary to keep the
> threads open to provide queries. This is too wasteful.
>
> thank you .
>
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by Shawn Heisey <el...@elyograg.org>.
On 3/10/2018 9:44 AM, Erick Erickson wrote:
> There are quite a number of reasons you may be seeing this, all having
> to do with trying to put too much stuff in too little hardware.
<snip>
> At any rate, there's no a-priori limit to the number of
> collections/replicas/whatever that Solr can deal with, the limits are
> your hardware.
Big +1 to everything Erick said. I was working on a response, but I
think the things that Erick said are better than what I was writing. I
did want to add a little bit of detail, though.
Let me throw some numbers at you:
With 900 collections that have 25 shards, the SolrCloud cluster is
handling 22500 individual indexes -- and that's for only one replica.
If I assume that you've only got one copy of all that data in Solr, then
every one of those 49 servers will average over 450 index cores! If you
planned for redundancy and replicationFactor is 2, then each server will
have over 900 index cores on it (45000 total indexes).
The index count numbers on each server might look like small numbers to
you ... but when you start understanding some of what Solr and Lucene
are doing under the covers, you start to realize that these numbers are
quite large.
If there's even a hint of significant index/query traffic going to those
collections, then those servers are going to be hard-pressed to keep up,
unless you've spent a LOT more money for hardware than what I see in a
typical server.
Have you ever heard of a phenomenon known as a performance knee? The
following link should open up to page 482 of the associated book.
Scroll up to page 481 and reading the two sections named "Hardware
Resource Requirements" and "Peak Load Behavior". Your reading would end
partway through page 482.
https://books.google.com/books?id=ka4QO9kXQFUC&pg=PA482&lpg=PA482&dq=performance+knee+scalability+graph&source=bl&ots=ytNn04PMU8&sig=r8GAGLIaT4mDwjCjYs_waQJbqo8&hl=en&sa=X&ved=0ahUKEwjH6bSdq-HZAhUJslMKHeQhALQQ6AEINDAB#v=onepage&q=performance%20knee%20scalability%20graph&f=false
A common tendency in the performance of computer systems/software is
that they show an increase in response time that's more or less linear
with load increases, until the system reaches a breaking point where the
smallest increase in load causes an incredibly large increase in
response times. That can lead to an outage situation where things take
so long that clients give up and go to an error state.
FYI: You're running a "dot zero" release. 6.0 is the first release
where MAJOR development changes had been accumulating that weren't in
5.x versions. The 6.x timeframe was an absolute hotbed of innovation on
SolrCloud.
There are typically more bugs in a .0 release than later minor
releases. If there are reasons for you to avoid running 7.x releases,
then you should upgrade to the latest 6.x. Right now this is 6.6.2, but
we are in the middle of the 6.6.3 release right now, and that is likely
to be announced any day now.
SolrCloud does not handle large numbers of collections very well,
especially if the shard and/or replica counts are high for each
collection. Up to a point, throwing a LOT of hardware at the problem
helps, but there are scalability issues related to the way SolrCloud
uses the ZooKeeper database. Eventually it just can't handle it. See
this issue for some experiments that I conducted on the problem:
https://issues.apache.org/jira/browse/SOLR-7191
Thanks,
Shawn
Re: The Impact of the Number of Collections on Indexing Performance
in Solr 6.0
Posted by Erick Erickson <er...@gmail.com>.
There are quite a number of reasons you may be seeing this, all having
to do with trying to put too much stuff in too little hardware.
Yes, an increasing number of collections will spin up an increasing
number of threads. Also your Jetty container has a limited thread pool
may be a factor. Or you may simply be starting to swap memory on/off
disk. Or you're hitting GC pauses. Or..... Given your statement that
speed decreases 50x, my suspicion is disk swapping, but that's a total
guess.
The key here is that you are hosting N replicas per JVM, and (maybe) M
JVMs per physical machine. It's unreasonable to assume that you can
keep adding more and more and more collections/replicas per JVM and
not eventually hit the limits of your hardware. And you mention that
indexing slows down, which leads me to assume that you're adding more
and more docs to each of these collections. This will eventually
simply blow up.
Here's a long blog on the topic:
https://lucidworks.com/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
There are two options that spring to mind:
1> get more hardware and distribute the replicas over more hardware
2> consider combining collections. So instead of having one collection
per user, say, have many users in the same collection and keep them
from seeing each other's data by adding the appropriate "fq" clause.
At any rate, there's no a-priori limit to the number of
collections/replicas/whatever that Solr can deal with, the limits are
your hardware.
Best,
Erick
On Fri, Mar 9, 2018 at 9:52 PM, 苗海泉 <ms...@gmail.com> wrote:
> hello,We found a problem. In solr 6.0, the indexing speed of solr is
> influenced by the number of solr collections. The speed is normal before
> the limit is reached. If the limit is reached, the indexing speed will
> decrease by 50 times.
>
> In our environment, there are 49 solr nodes. If each collection has 25
> shards, you can maintain high-speed indexing until the total number of
> collections is about 900. To reduce the number of collections to the limit,
> the speed will increase. Go up.
> If each collection is 49 shards, the total number of collections can only
> be about 700, exceeding this value will cause the index to drop
> dramatically.
> In the explanation, we are single copies, and multiple copies will cause
> serious stability problems in the large solr cluster environment.
>
> At first I suspect that it was due to too many thread submissions, and
> there are still problems with this method, so I'm inclined to
> searcherExecutor thread pool thread. This is just my guess, I want to know
> the real reason. Can someone know if I can help?
>
> Also, I noticed that the searcherExecutor thread and solr collection's
> shards basically correspond to each other. How can I reduce the number of
> threads or even close it? Although there are many collections in our
> environment, there are few queries and it is not necessary to keep the
> threads open to provide queries. This is too wasteful.
>
> thank you .