You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Natarajan, Rajeswari" <ra...@sap.com.INVALID> on 2022/10/11 18:30:30 UTC
Not able to Create/Delete collection
Hi,
We have a six node solrcloud cluster and we have about 780 collections each having one shard and 3 replicas. We have a situation where now create/delete collection times out and when we try async option , the job gets submitted and remains like that for hours. Tried restarting the solr nodes ,nothing changed. CPU (~0%) usage and heap (<70%) are good .
In the solradmin UI , cloud ->Tree section takes long time , looks like it takes times to connect to zk. Thinking of deleting collections manually in disk and in zk . Is there any other solution to get around this issue , don’t see any error in the logs.
Any thoughts? . Please let me know.
Thanks,
Rajeswari
Re: Not able to Create/Delete collection
Posted by Shawn Heisey <el...@elyograg.org>.
On 10/11/22 14:33, Natarajan, Rajeswari wrote:
> Thank you for the reply. There was a typo in my email , we have 6 node cluster. So it does not matter how many nodes we have in a cluster, the collection limit for a solrcloud is fixed?.
There isn't a hard limit. But when you get more than a few hundred
collections, the amount of work required with the database in ZooKeeper
becomes very large and runs a serious risk of timing out. Putting the
ZK database on VERY fast disks like SSD or RAID10 with lots of drives
might help with that.
The number of nodes seem to not affect this problem. Or at least the
effect is very small. There is a possibility that if you have a very
large number of nodes such that each node only has to worry about a
handful of cores, it might behave better. But I do not have the
infrastructure to test that theory.
Thanks,
Shawn
Re: Not able to Create/Delete collection
Posted by "Natarajan, Rajeswari" <ra...@sap.com.INVALID>.
Thank you for the reply. There was a typo in my email , we have 6 node cluster. So it does not matter how many nodes we have in a cluster, the collection limit for a solrcloud is fixed?.
Are there plans to fix this potential problem , as multiple solrclouds would be needed to support many solr collections.
Regards,
Rajeswari
On 10/11/22, 12:53 PM, "Shawn Heisey" <el...@elyograg.org> wrote:
On 10/11/22 12:30, Natarajan, Rajeswari wrote:
> We have a six node solrcloud cluster and we have about 780 collections each having one shard and 3 replicas. We have a situation where now create/delete collection times out and when we try async option , the job gets submitted and remains like that for hours. Tried restarting the solr nodes ,nothing changed. CPU (~0%) usage and heap (<70%) are good .
>
> In the solradmin UI , cloud ->Tree section takes long time , looks like it takes times to connect to zk. Thinking of deleting collections manually in disk and in zk . Is there any other solution to get around this issue , don’t see any error in the logs.
That many collections will lead to problems. SolrCloud has a
scalability problem when the number of collections gets beyond a few
hundred. I did some investigation into this a while back.
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSOLR-7191&data=05%7C01%7Crajeswari.natarajan%40sap.com%7Ca389d396798445eedf8308daabc25010%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638011148325033593%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IyJseI1TCFFRM6VtA6RdKL2bltoQ0NK3n%2BHHInkA0O8%3D&reserved=0
This issue was marked as resolved, though no code was committed in
connection with the issue. Later tests that I did suggest that the
problem has gotten worse, not better, since version 6.x. It wasn't a
rigorous re-test, so I have no hard numbers.
Thanks,
Shawn
Re: Not able to Create/Delete collection
Posted by Shawn Heisey <el...@elyograg.org>.
On 10/11/22 12:30, Natarajan, Rajeswari wrote:
> We have a six node solrcloud cluster and we have about 780 collections each having one shard and 3 replicas. We have a situation where now create/delete collection times out and when we try async option , the job gets submitted and remains like that for hours. Tried restarting the solr nodes ,nothing changed. CPU (~0%) usage and heap (<70%) are good .
>
> In the solradmin UI , cloud ->Tree section takes long time , looks like it takes times to connect to zk. Thinking of deleting collections manually in disk and in zk . Is there any other solution to get around this issue , don’t see any error in the logs.
That many collections will lead to problems. SolrCloud has a
scalability problem when the number of collections gets beyond a few
hundred. I did some investigation into this a while back.
https://issues.apache.org/jira/browse/SOLR-7191
This issue was marked as resolved, though no code was committed in
connection with the issue. Later tests that I did suggest that the
problem has gotten worse, not better, since version 6.x. It wasn't a
rigorous re-test, so I have no hard numbers.
Thanks,
Shawn
Re: Not able to Create/Delete collection
Posted by "Natarajan, Rajeswari" <ra...@sap.com.INVALID>.
We are using solr 8.11.1 and ZK version 3.8.0
Thanks
From: "Natarajan, Rajeswari" <ra...@sap.com>
Date: Tuesday, October 11, 2022 at 11:30 AM
To: "users@solr.apache.org" <us...@solr.apache.org>
Subject: Not able to Create/Delete collection
Hi,
We have a six node solrcloud cluster and we have about 780 collections each having one shard and 3 replicas. We have a situation where now create/delete collection times out and when we try async option , the job gets submitted and remains like that for hours. Tried restarting the solr nodes ,nothing changed. CPU (~0%) usage and heap (<70%) are good .
In the solradmin UI , cloud ->Tree section takes long time , looks like it takes times to connect to zk. Thinking of deleting collections manually in disk and in zk . Is there any other solution to get around this issue , don’t see any error in the logs.
Any thoughts? . Please let me know.
Thanks,
Rajeswari