You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Karl Stoney <Ka...@autotrader.co.uk.INVALID> on 2020/02/13 09:34:05 UTC

Async RELOADCOLLECTION never completes

Hi,
We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

❯ curl -s http://solr.search-solr.prod.k8.atcloud.io/solr/admin/collections\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
{
  "responseHeader": {
    "status": 0,
    "QTime": 2
  },
  "status": {
    "state": "running",
    "msg": "found [1581585716] in running tasks"
  }
}

The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

The only way I’ve found to clear this up is to rolling restart solr.

Solr 8.4.1

Any ideas?
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

Re: Async RELOADCOLLECTION never completes

Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.

I actually narrowed this down to changing the schema version from 1.5 to 1.6 and then doing a RELOADCOLLECTION, it consistently hangs.  Several of our nodes go briefly into a recovering state too.
________________________________
From: Karl Stoney <Ka...@autotrader.co.uk.INVALID>
Sent: 13 February 2020 09:49
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Re: Async RELOADCOLLECTION never completes

When performing a rolling restart we see:

09:43:31.890 [OverseerThreadFactory-42-thread-5-processing-n:solr-5.search-solr.prod.k8.atcloud.io:80_solr] ERROR org.apache.solr.cloud.OverseerTaskProcessor - :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-map-failure

Which I find interesting, everything (resources wise) is very healthy.

On 13/02/2020, 09:34, "Karl Stoney" <Ka...@autotrader.co.uk.INVALID> wrote:

    Hi,
    We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

    ❯ curl -s https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr.search-solr.prod.k8.atcloud.io%2Fsolr%2Fadmin%2Fcollections&amp;data=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C7a5d7c308dac49b5ab3408d7b06a0ff3%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637171841881478917&amp;sdata=fjl0QkOCCanmgUHAXX%2F5HvzBcexCmugLItWNIzG1D%2Fc%3D&amp;reserved=0\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
    {
      "responseHeader": {
        "status": 0,
        "QTime": 2
      },
      "status": {
        "state": "running",
        "msg": "found [1581585716] in running tasks"
      }
    }

    The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

    The only way I’ve found to clear this up is to rolling restart solr.

    Solr 8.4.1

    Any ideas?
    This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

Re: Async RELOADCOLLECTION never completes

Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.

When performing a rolling restart we see:

09:43:31.890 [OverseerThreadFactory-42-thread-5-processing-n:solr-5.search-solr.prod.k8.atcloud.io:80_solr] ERROR org.apache.solr.cloud.OverseerTaskProcessor - :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-map-failure

Which I find interesting, everything (resources wise) is very healthy.

On 13/02/2020, 09:34, "Karl Stoney" <Ka...@autotrader.co.uk.INVALID> wrote:

Hi,
We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

❯ curl -s https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr.search-solr.prod.k8.atcloud.io%2Fsolr%2Fadmin%2Fcollections&amp;data=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C3a627213825a4b56415008d7b067eb73%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637171832681589729&amp;sdata=Kx3OY%2BMkXw%2Bob0M0ZRmnehfAxffoSdGvJyV%2FlfdwdxY%3D&amp;reserved=0\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
{
"responseHeader": {
"status": 0,
"QTime": 2
},
"status": {
"state": "running",
"msg": "found [1581585716] in running tasks"
}
}

The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

The only way I’ve found to clear this up is to rolling restart solr.

Solr 8.4.1

Any ideas?
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.