You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Kelly, Frank" <fr...@here.com> on 2017/09/11 20:38:37 UTC
SolrCloud 5.3.1 "IndexWriter is closed"
Just wondering if anyone has seen this before and might understand why this is happening
When we deploy a new schema.xml adding some new search terms we get the dreaded “IndexWriter is closed” exception and the only solution we have found to date is a Solr restart :-(
Environment:
* Solr 5.3.1 in Solr Cloud (3 shards each with 3 replicas across 3 EC2 Vms)
* 300m+ documents (60+ GB index)
* Relevant settings?
* SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=1000”
* SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=300000"
Is there anything “unsafe” about reload on a collection that is handling live traffic?
Could be related to https://issues.apache.org/jira/browse/SOLR-9830
And https://issues.apache.org/jira/browse/SOLR-9699 in particular?
Cheers!
-Frank
[Description: Macintosh HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]
Frank Kelly
Principal Software Engineer
Identity Profile Team (SCBE, Traces, CDA)
HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
[Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]<http://360.here.com/> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif] <https://www.twitter.com/here> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif] <https://www.facebook.com/here> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif] <https://www.linkedin.com/company/heremaps> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif] <https://www.instagram.com/here/>
Re: SolrCloud 5.3.1 "IndexWriter is closed"
Posted by Susheel Kumar <su...@gmail.com>.
During Reload collection, it does close the IndexWriter and opens a new.
So looks like with live traffic (is it heavy traffic, how many qps?) and
collection reload, its running into a situation when IndexWriter is not
able to open up again. Not sure if this issue might have been fixed in 5
or 6.x and may work fine in latest 6.6.1 if you could test anyway.
https://www.quora.com/What-happens-during-a-Solr-Core-Reload
On Tue, Sep 12, 2017 at 12:47 PM, Kelly, Frank <fr...@here.com> wrote:
> The schema change doesn¹t seem to be making any difference - just the act
> of a reload whilst handling live traffic.
>
> The reload takes about 30 seconds and soon after (within a few seconds) we
> start to see IndexWriter closed exceptions
>
> -Frank
>
>
> Frank Kelly
> Principal Software Engineer
> Identity Profile Team (SCBE, Traces, CDA)
>
> HERE
> 5 Wayside Rd, Burlington, MA 01803, USA
> 42° 29' 7" N 71° 11' 32" W
>
> <http://360.here.com/> <https://www.twitter.com/here>
> <https://www.facebook.com/here>
> <https://www.linkedin.com/company/heremaps>
> <https://www.instagram.com/here/>
>
>
>
> On 9/12/17, 9:13 AM, "Susheel Kumar" <su...@gmail.com> wrote:
>
> >Kelly -
> >
> >If you do not make any change to schema and just reload your collection,
> >does it work fine? How much time it takes to reload the collection?
> >
> >I am suspecting some conflict with commit frequency (5mins) and collection
> >reload.
> >
> >Thnx
>
>
Re: SolrCloud 5.3.1 "IndexWriter is closed"
Posted by "Kelly, Frank" <fr...@here.com>.
The schema change doesn¹t seem to be making any difference - just the act
of a reload whilst handling live traffic.
The reload takes about 30 seconds and soon after (within a few seconds) we
start to see IndexWriter closed exceptions
-Frank
Frank Kelly
Principal Software Engineer
Identity Profile Team (SCBE, Traces, CDA)
HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
<http://360.here.com/> <https://www.twitter.com/here>
<https://www.facebook.com/here>
<https://www.linkedin.com/company/heremaps>
<https://www.instagram.com/here/>
On 9/12/17, 9:13 AM, "Susheel Kumar" <su...@gmail.com> wrote:
>Kelly -
>
>If you do not make any change to schema and just reload your collection,
>does it work fine? How much time it takes to reload the collection?
>
>I am suspecting some conflict with commit frequency (5mins) and collection
>reload.
>
>Thnx
Re: SolrCloud 5.3.1 "IndexWriter is closed"
Posted by Susheel Kumar <su...@gmail.com>.
Kelly -
If you do not make any change to schema and just reload your collection,
does it work fine? How much time it takes to reload the collection?
I am suspecting some conflict with commit frequency (5mins) and collection
reload.
Thnx
On Tue, Sep 12, 2017 at 6:59 AM, Kelly, Frank <fr...@here.com> wrote:
> No - these are new terms for new documents we will be adding later so no
> need to reindex old documents.
>
> Frank
>
>
> Frank Kelly
> Principal Software Engineer
> Identity Profile Team (SCBE, Traces, CDA)
>
> HERE
> 5 Wayside Rd, Burlington, MA 01803, USA
> 42° 29' 7" N 71° 11' 32" W
>
> <http://360.here.com/> <https://www.twitter.com/here>
> <https://www.facebook.com/here>
> <https://www.linkedin.com/company/heremaps>
> <https://www.instagram.com/here/>
>
>
>
> On 9/12/17, 6:27 AM, "Rick Leir" <rl...@leirtech.com> wrote:
>
> >Frank,
> >
> >I assume you re-index everything after changing schema.xml?
> >
> >cheers -- Rick
> >
>
>
Re: SolrCloud 5.3.1 "IndexWriter is closed"
Posted by "Kelly, Frank" <fr...@here.com>.
No - these are new terms for new documents we will be adding later so no
need to reindex old documents.
Frank
Frank Kelly
Principal Software Engineer
Identity Profile Team (SCBE, Traces, CDA)
HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
<http://360.here.com/> <https://www.twitter.com/here>
<https://www.facebook.com/here>
<https://www.linkedin.com/company/heremaps>
<https://www.instagram.com/here/>
On 9/12/17, 6:27 AM, "Rick Leir" <rl...@leirtech.com> wrote:
>Frank,
>
>I assume you re-index everything after changing schema.xml?
>
>cheers -- Rick
>
Re: SolrCloud 5.3.1 "IndexWriter is closed"
Posted by Rick Leir <rl...@leirtech.com>.
Frank,
I assume you re-index everything after changing schema.xml?
cheers -- Rick
On 2017-09-11 04:38 PM, Kelly, Frank wrote:
> Just wondering if anyone has seen this before and might understand why
> this is happening
>
> When we deploy a new schema.xml adding some new search terms we get
> the dreaded “IndexWriter is closed” exception and the only solution we
> have found to date is a Solr restart :-(
>
> Environment:
>
> * Solr 5.3.1 in Solr Cloud (3 shards each with 3 replicas across 3
> EC2 Vms)
> * 300m+ documents (60+ GB index)
> * Relevant settings?
> o SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=1000”
> o SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=300000"
>
>
> Is there anything “unsafe” about reload on a collection that is
> handling live traffic?
> Could be related to https://issues.apache.org/jira/browse/SOLR-9830
> And https://issues.apache.org/jira/browse/SOLR-9699 in particular?
>
> Cheers!
>
> -Frank
>
> Description: Macintosh
> HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf
>
> *Frank Kelly*
>
> *Principal Software Engineer*
>
> Identity Profile Team (SCBE, Traces, CDA)
>
>
> HERE
>
> 5 Wayside Rd, Burlington, MA 01803, USA
>
> *42° 29' 7" N 71° 11' 32" W*
>
> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif
> <http://360.here.com/> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif
> <https://www.twitter.com/here> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif
> <https://www.facebook.com/here> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif
> <https://www.linkedin.com/company/heremaps> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif
> <https://www.instagram.com/here/>