Schema update/Reload during Live traffic

[Susheel Kumar <su...@gmail.com>]

Hello,

I did schema update to Solr cloud of Source CDCR cluster and same on
target. After Collection Reload, noticed "error opening searcher" /
IndexWriter closed etc. on leader node while all replica's went into
recovery mode.

Later after restarting Solr on Leader noticed below too many file open
errors.  The ulimit shows unlimited.

What could have gone wrong above/and how can we avoid.  This is with 6.6.2
and there were ingestions/delete and searches happening the time I reload
the collection.

Caused by: org.apache.solr.common.SolrException:
java.io.FileNotFoundException:
/app/solr/data/COLL_shard1_replica4/data/tlog/tlog.0000000000000002214.1591528549576081408
(Too many open files)

        at
org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:250)

        at
org.apache.solr.update.CdcrTransactionLog.incref(CdcrTransactionLog.java:179)

        at
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1342)

        at org.apache.solr.update.UpdateLog.init(UpdateLog.java:393)

        at org.apache.solr.update.CdcrUpdateLog.init(CdcrUpdateLog.java:77)

        at
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:153)

        at
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:110)

        at
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:760)

        at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:822)

        at
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)

        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:947)

Re: Schema update/Reload during Live traffic

[Shawn Heisey <ap...@elyograg.org>]

On 3/23/2018 10:32 PM, Susheel Kumar wrote:
> I did schema update to Solr cloud of Source CDCR cluster and same on
> target. After Collection Reload, noticed "error opening searcher" /
> IndexWriter closed etc. on leader node while all replica's went into
> recovery mode.
>
> Later after restarting Solr on Leader noticed below too many file open
> errors.  The ulimit shows unlimited.
>
> What could have gone wrong above/and how can we avoid.  This is with 6.6.2
> and there were ingestions/delete and searches happening the time I reload
> the collection.
>
> Caused by: org.apache.solr.common.SolrException:
> java.io.FileNotFoundException:
> /app/solr/data/COLL_shard1_replica4/data/tlog/tlog.0000000000000002214.1591528549576081408
> (Too many open files)

A 'ulimit' command that you typed may have shown unlimited, but the Solr 
process apparently DOES have a limit.

If you are trying to change limits with the ulimit command for Solr, I 
have frequently seen this NOT work.  On Linux, /etc/security/limits.conf 
appears to be the only way to successfully increase the open file 
limit.  The default limit on Linux is normally 1024, which is pretty low.

I would recommend not doing reloads when there is a lot of indexing 
activity.  When indexing is heavy, it can lead to recovery operations, 
if the number of documents indexed during the reload exceeds the limit 
for syncing from transaction logs.  This limit is normally 100 docs.

Thanks,
Shawn