Solr Collection reload task has been in running state for a very long time

[Moulay Hicham <ma...@gmail.com>]

Hi,

We have a solr cluster of 30 nodes with a Replication Factor =3.
Each index size is about 80GB.
Solr version is 8.1
The cluster has high TPS both in read and write.

We have recently made a schema change and uploaded it using ZKCLI
script. Then we issue a collection reload async request:
admin/collections?action=RELOAD&name=<collectionname>&async=1000'

When we check on the status of this request, it shows that it's still running:

admin/collections?action=REQUESTSTATUS&requestid=1000'
{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "status":{
    "state":"running",
    "msg":"found [1000] in running tasks"}}

This task has been in a running state for about 5 hours so far. I am
not sure if this is expected or the status of this task failed or
completed but never reported back to zookeeper.

Also if running for that long - is it because the index is being
actively (with high TPS) updated? We have a softcommit of 10s and
hadcommit of 60s.

Please help me understand what's going on.

Thanks,
Moulay

Re: Solr Collection reload task has been in running state for a very long time

[Moulay Hicham <ma...@gmail.com>]

Hi,

I will really appreciate if someone can help me with this.

Thank you,

Moulay

On Thu, Dec 10, 2020, 8:28 AM Moulay Hicham <ma...@gmail.com> wrote:

> Hi,
>
> We have a solr cluster of 30 nodes with a Replication Factor =3.
> Each index size is about 80GB.
> Solr version is 8.1
> The cluster has high TPS both in read and write.
>
> We have recently made a schema change and uploaded it using ZKCLI
> script. Then we issue a collection reload async request:
> admin/collections?action=RELOAD&name=<collectionname>&async=1000'
>
> When we check on the status of this request, it shows that it's still
> running:
>
> admin/collections?action=REQUESTSTATUS&requestid=1000'
> {
>   "responseHeader":{
>     "status":0,
>     "QTime":1},
>   "status":{
>     "state":"running",
>     "msg":"found [1000] in running tasks"}}
>
> This task has been in a running state for about 5 hours so far. I am
> not sure if this is expected or the status of this task failed or
> completed but never reported back to zookeeper.
>
> Also if running for that long - is it because the index is being
> actively (with high TPS) updated? We have a softcommit of 10s and
> hadcommit of 60s.
>
> Please help me understand what's going on.
>
> Thanks,
> Moulay
>