You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hakim Benoudjit <h....@gmail.com> on 2014/05/04 14:06:51 UTC
Core failure when a lot of processes are indexing
Hi guys,
I have a lot of scripts running *concurrently *& indexing on the same index
(*collection*). This throws the following error:
*SolrCore 'collection2' is not available due to init failure: Error opening
new searcher</str><str name="trace">org.apache.solr.common.SolrException:
SolrCore 'collection2' is not available due to init failure: Error opening
new searcher*
Which from what I've understood means that the index has been *corrupted*.
So what is the solution that you can suggest me to use? Active Messagin
(ActiveMQ or RabbitMQ) maybe?
*PS: *
-- By adding a *sleep() *in each of these scripts, the index managed to not
be corrupted, that's why I'm suspecting that the number of concurrent
request (*update*) received by solr, are causing this error.
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Hakim Benoudjit <h....@gmail.com>.
Thanks Erick for the explanation.
I'll set my autocommit max time to 30 seconds then.
But, I can let soft commit max time to 1/4 hour since it's an ads plateform
which needs to be updated regularly.
2014-05-05 21:14 GMT+01:00 Erick Erickson <er...@gmail.com>:
> Take a look through the article I linked, 5 minutes may be an issue
> since the transaction log will hold all 5 minutes worth of input. In
> batch processes this can be quite a bit of data. Worse, when a Solr
> instance terminates unexpectedly, the entire transaction log can be
> replayed.
>
> Consider setting your autommit max time to something much shorter, say
> 30 seconds. Or even less. NOTE openSearcher should be false.
>
> Then set your soft commit time to the latency you can stand, i.e. if
> the users don't need to be able to search for a long time you can set
> this to hours.
>
> FWIW,
> Erick
>
> On Mon, May 5, 2014 at 11:03 AM, Hakim Benoudjit <h....@gmail.com>
> wrote:
> > I've tried it & it worked by letting solr do the commit instead of my
> solr
> > client.
> > In solrconfig.xml:
> > autocommit max_time has been set to 5 minutes & autosoftcommit max_time
> to
> > something bigger.
> >
> > Thanks a lot guys!
> >
> >
> > 2014-05-05 16:30 GMT+01:00 Erick Erickson <er...@gmail.com>:
> >
> >> You should not be committing from the client by and large, use the
> >> <autoCommit> and <autoSoftCommit> options in solrconfig.xml.
> >>
> >> See:
> >>
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h....@gmail.com>
> >> wrote:
> >> > Is there an option in Solr (solrconfig.xml or somewhere else) to
> >> regularize
> >> > commits to the index.
> >> > I meant to do a 'sleep' between each commit to the index, when data
> >> > to-be-indexed is waiting inside a stack.
> >> >
> >> >
> >> > 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h....@gmail.com>:
> >> >
> >> >> The index is made with the same version of solr, that is searching
> >> >> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same
> too.
> >> >> The only way for me to solve this issue is to let only one process to
> >> >> index at the same time. Wouldnt a layer of message queue resolve this
> >> issue?
> >> >>
> >> >>
> >> >> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
> >> >>
> >> >> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
> >> >>> > Ok. These files contain what you've requested:
> >> >>> >
> >> >>> > First (the xml error): http://pastebin.com/ZcagK3T7
> >> >>> > Second (java params): http://pastebin.com/JtWQpp6s
> >> >>> > Third (Solr version): http://pastebin.com/wYdpdsAW
> >> >>>
> >> >>> Are you running with an index originally built by an earlier
> version of
> >> >>> Solr? If you are, you may be running into a known bug. The last
> >> >>> "caused by" section of the java stacktrace looks similar to the one
> in
> >> >>> this issue -- which is indeed index corruption:
> >> >>>
> >> >>> https://issues.apache.org/jira/browse/LUCENE-5377
> >> >>>
> >> >>> If that's the problem you're experiencing, upgrading your Solr
> version
> >> >>> will hopefully fix it. Simply dropping in the 4.6.1 war file and
> any
> >> >>> contrib jars should cause zero problems for your 4.6.0 install.
> >> >>> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
> >> >>>
> >> >>> Thanks,
> >> >>> Shawn
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> Hakim Benoudjit.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Hakim Benoudjit.
> >>
> >
> >
> >
> > --
> > Hakim Benoudjit.
>
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Erick Erickson <er...@gmail.com>.
Take a look through the article I linked, 5 minutes may be an issue
since the transaction log will hold all 5 minutes worth of input. In
batch processes this can be quite a bit of data. Worse, when a Solr
instance terminates unexpectedly, the entire transaction log can be
replayed.
Consider setting your autommit max time to something much shorter, say
30 seconds. Or even less. NOTE openSearcher should be false.
Then set your soft commit time to the latency you can stand, i.e. if
the users don't need to be able to search for a long time you can set
this to hours.
FWIW,
Erick
On Mon, May 5, 2014 at 11:03 AM, Hakim Benoudjit <h....@gmail.com> wrote:
> I've tried it & it worked by letting solr do the commit instead of my solr
> client.
> In solrconfig.xml:
> autocommit max_time has been set to 5 minutes & autosoftcommit max_time to
> something bigger.
>
> Thanks a lot guys!
>
>
> 2014-05-05 16:30 GMT+01:00 Erick Erickson <er...@gmail.com>:
>
>> You should not be committing from the client by and large, use the
>> <autoCommit> and <autoSoftCommit> options in solrconfig.xml.
>>
>> See:
>> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> Best,
>> Erick
>>
>> On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h....@gmail.com>
>> wrote:
>> > Is there an option in Solr (solrconfig.xml or somewhere else) to
>> regularize
>> > commits to the index.
>> > I meant to do a 'sleep' between each commit to the index, when data
>> > to-be-indexed is waiting inside a stack.
>> >
>> >
>> > 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h....@gmail.com>:
>> >
>> >> The index is made with the same version of solr, that is searching
>> >> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same too.
>> >> The only way for me to solve this issue is to let only one process to
>> >> index at the same time. Wouldnt a layer of message queue resolve this
>> issue?
>> >>
>> >>
>> >> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
>> >>
>> >> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
>> >>> > Ok. These files contain what you've requested:
>> >>> >
>> >>> > First (the xml error): http://pastebin.com/ZcagK3T7
>> >>> > Second (java params): http://pastebin.com/JtWQpp6s
>> >>> > Third (Solr version): http://pastebin.com/wYdpdsAW
>> >>>
>> >>> Are you running with an index originally built by an earlier version of
>> >>> Solr? If you are, you may be running into a known bug. The last
>> >>> "caused by" section of the java stacktrace looks similar to the one in
>> >>> this issue -- which is indeed index corruption:
>> >>>
>> >>> https://issues.apache.org/jira/browse/LUCENE-5377
>> >>>
>> >>> If that's the problem you're experiencing, upgrading your Solr version
>> >>> will hopefully fix it. Simply dropping in the 4.6.1 war file and any
>> >>> contrib jars should cause zero problems for your 4.6.0 install.
>> >>> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
>> >>>
>> >>> Thanks,
>> >>> Shawn
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Hakim Benoudjit.
>> >>
>> >
>> >
>> >
>> > --
>> > Hakim Benoudjit.
>>
>
>
>
> --
> Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Hakim Benoudjit <h....@gmail.com>.
I've tried it & it worked by letting solr do the commit instead of my solr
client.
In solrconfig.xml:
autocommit max_time has been set to 5 minutes & autosoftcommit max_time to
something bigger.
Thanks a lot guys!
2014-05-05 16:30 GMT+01:00 Erick Erickson <er...@gmail.com>:
> You should not be committing from the client by and large, use the
> <autoCommit> and <autoSoftCommit> options in solrconfig.xml.
>
> See:
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Best,
> Erick
>
> On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h....@gmail.com>
> wrote:
> > Is there an option in Solr (solrconfig.xml or somewhere else) to
> regularize
> > commits to the index.
> > I meant to do a 'sleep' between each commit to the index, when data
> > to-be-indexed is waiting inside a stack.
> >
> >
> > 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h....@gmail.com>:
> >
> >> The index is made with the same version of solr, that is searching
> >> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same too.
> >> The only way for me to solve this issue is to let only one process to
> >> index at the same time. Wouldnt a layer of message queue resolve this
> issue?
> >>
> >>
> >> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
> >>
> >> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
> >>> > Ok. These files contain what you've requested:
> >>> >
> >>> > First (the xml error): http://pastebin.com/ZcagK3T7
> >>> > Second (java params): http://pastebin.com/JtWQpp6s
> >>> > Third (Solr version): http://pastebin.com/wYdpdsAW
> >>>
> >>> Are you running with an index originally built by an earlier version of
> >>> Solr? If you are, you may be running into a known bug. The last
> >>> "caused by" section of the java stacktrace looks similar to the one in
> >>> this issue -- which is indeed index corruption:
> >>>
> >>> https://issues.apache.org/jira/browse/LUCENE-5377
> >>>
> >>> If that's the problem you're experiencing, upgrading your Solr version
> >>> will hopefully fix it. Simply dropping in the 4.6.1 war file and any
> >>> contrib jars should cause zero problems for your 4.6.0 install.
> >>> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
> >>>
> >>> Thanks,
> >>> Shawn
> >>>
> >>>
> >>
> >>
> >> --
> >> Hakim Benoudjit.
> >>
> >
> >
> >
> > --
> > Hakim Benoudjit.
>
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Erick Erickson <er...@gmail.com>.
You should not be committing from the client by and large, use the
<autoCommit> and <autoSoftCommit> options in solrconfig.xml.
See: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Best,
Erick
On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h....@gmail.com> wrote:
> Is there an option in Solr (solrconfig.xml or somewhere else) to regularize
> commits to the index.
> I meant to do a 'sleep' between each commit to the index, when data
> to-be-indexed is waiting inside a stack.
>
>
> 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h....@gmail.com>:
>
>> The index is made with the same version of solr, that is searching
>> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same too.
>> The only way for me to solve this issue is to let only one process to
>> index at the same time. Wouldnt a layer of message queue resolve this issue?
>>
>>
>> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
>>
>> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
>>> > Ok. These files contain what you've requested:
>>> >
>>> > First (the xml error): http://pastebin.com/ZcagK3T7
>>> > Second (java params): http://pastebin.com/JtWQpp6s
>>> > Third (Solr version): http://pastebin.com/wYdpdsAW
>>>
>>> Are you running with an index originally built by an earlier version of
>>> Solr? If you are, you may be running into a known bug. The last
>>> "caused by" section of the java stacktrace looks similar to the one in
>>> this issue -- which is indeed index corruption:
>>>
>>> https://issues.apache.org/jira/browse/LUCENE-5377
>>>
>>> If that's the problem you're experiencing, upgrading your Solr version
>>> will hopefully fix it. Simply dropping in the 4.6.1 war file and any
>>> contrib jars should cause zero problems for your 4.6.0 install.
>>> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
>>
>> --
>> Hakim Benoudjit.
>>
>
>
>
> --
> Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Hakim Benoudjit <h....@gmail.com>.
Is there an option in Solr (solrconfig.xml or somewhere else) to regularize
commits to the index.
I meant to do a 'sleep' between each commit to the index, when data
to-be-indexed is waiting inside a stack.
2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h....@gmail.com>:
> The index is made with the same version of solr, that is searching
> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same too.
> The only way for me to solve this issue is to let only one process to
> index at the same time. Wouldnt a layer of message queue resolve this issue?
>
>
> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
>
> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
>> > Ok. These files contain what you've requested:
>> >
>> > First (the xml error): http://pastebin.com/ZcagK3T7
>> > Second (java params): http://pastebin.com/JtWQpp6s
>> > Third (Solr version): http://pastebin.com/wYdpdsAW
>>
>> Are you running with an index originally built by an earlier version of
>> Solr? If you are, you may be running into a known bug. The last
>> "caused by" section of the java stacktrace looks similar to the one in
>> this issue -- which is indeed index corruption:
>>
>> https://issues.apache.org/jira/browse/LUCENE-5377
>>
>> If that's the problem you're experiencing, upgrading your Solr version
>> will hopefully fix it. Simply dropping in the 4.6.1 war file and any
>> contrib jars should cause zero problems for your 4.6.0 install.
>> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
>>
>> Thanks,
>> Shawn
>>
>>
>
>
> --
> Hakim Benoudjit.
>
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Hakim Benoudjit <h....@gmail.com>.
The index is made with the same version of solr, that is searching (4.6.0),
the config file (solrconfig.xml) & schema.xml is the same too.
The only way for me to solve this issue is to let only one process to index
at the same time. Wouldnt a layer of message queue resolve this issue?
2014-05-04 18:33 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
> > Ok. These files contain what you've requested:
> >
> > First (the xml error): http://pastebin.com/ZcagK3T7
> > Second (java params): http://pastebin.com/JtWQpp6s
> > Third (Solr version): http://pastebin.com/wYdpdsAW
>
> Are you running with an index originally built by an earlier version of
> Solr? If you are, you may be running into a known bug. The last
> "caused by" section of the java stacktrace looks similar to the one in
> this issue -- which is indeed index corruption:
>
> https://issues.apache.org/jira/browse/LUCENE-5377
>
> If that's the problem you're experiencing, upgrading your Solr version
> will hopefully fix it. Simply dropping in the 4.6.1 war file and any
> contrib jars should cause zero problems for your 4.6.0 install.
> Upgrading to 4.7.2 or 4.8.0 should be done with more care.
>
> Thanks,
> Shawn
>
>
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/4/2014 9:30 AM, Hakim Benoudjit wrote:
> Ok. These files contain what you've requested:
>
> First (the xml error): http://pastebin.com/ZcagK3T7
> Second (java params): http://pastebin.com/JtWQpp6s
> Third (Solr version): http://pastebin.com/wYdpdsAW
Are you running with an index originally built by an earlier version of
Solr? If you are, you may be running into a known bug. The last
"caused by" section of the java stacktrace looks similar to the one in
this issue -- which is indeed index corruption:
https://issues.apache.org/jira/browse/LUCENE-5377
If that's the problem you're experiencing, upgrading your Solr version
will hopefully fix it. Simply dropping in the 4.6.1 war file and any
contrib jars should cause zero problems for your 4.6.0 install.
Upgrading to 4.7.2 or 4.8.0 should be done with more care.
Thanks,
Shawn
Re: Core failure when a lot of processes are indexing
Posted by Hakim Benoudjit <h....@gmail.com>.
Ok. These files contain what you've requested:
First (the xml error): http://pastebin.com/ZcagK3T7
Second (java params): http://pastebin.com/JtWQpp6s
Third (Solr version): http://pastebin.com/wYdpdsAW
2014-05-04 14:23 GMT+01:00 Shawn Heisey <so...@elyograg.org>:
> On 5/4/2014 6:06 AM, Hakim Benoudjit wrote:
> > I have a lot of scripts running *concurrently *& indexing on the same
> index
> > (*collection*). This throws the following error:
> >
> > *SolrCore 'collection2' is not available due to init failure: Error
> opening
> > new searcher</str><str name="trace">org.apache.solr.common.SolrException:
> > SolrCore 'collection2' is not available due to init failure: Error
> opening
> > new searcher*
> >
> > Which from what I've understood means that the index has been
> *corrupted*.
>
> This can mean a *lot* of things. Index corruption is one of the last
> things I would suspect. It can happen, but it is highly unlikely.
>
> You would need to include the full error message (which could be more
> than one line) and the java stacktrace (which will be MANY lines). This
> may mean you need to find the logfile, not something that shows up in
> the admin UI. We'd also need to know the Solr version and as many other
> details about your server, java, Solr startup, and index as you can
> include. The config and schema would be great to have. Note that the
> mailing list does not deal well with attachments, so anything like
> screenshots and configs will generally need to be hosted elsewhere and
> linked.
>
> > So what is the solution that you can suggest me to use? Active Messagin
> > (ActiveMQ or RabbitMQ) maybe?
> >
> > *PS: *
> > -- By adding a *sleep() *in each of these scripts, the index managed to
> not
> > be corrupted, that's why I'm suspecting that the number of concurrent
> > request (*update*) received by solr, are causing this error.
>
> Solr is fully threadsafe. It can handle multiple concurrent requests.
> There was a relatively recent problem with multiple concurrent indexing
> requests against Solr when running in SolrCloud mode with many shards,
> but as far as I know this did not lead to problems starting a searcher,
> it caused indexing to hang and not proceed.
>
> Thanks,
> Shawn
>
>
--
Hakim Benoudjit.
Re: Core failure when a lot of processes are indexing
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/4/2014 6:06 AM, Hakim Benoudjit wrote:
> I have a lot of scripts running *concurrently *& indexing on the same index
> (*collection*). This throws the following error:
>
> *SolrCore 'collection2' is not available due to init failure: Error opening
> new searcher</str><str name="trace">org.apache.solr.common.SolrException:
> SolrCore 'collection2' is not available due to init failure: Error opening
> new searcher*
>
> Which from what I've understood means that the index has been *corrupted*.
This can mean a *lot* of things. Index corruption is one of the last
things I would suspect. It can happen, but it is highly unlikely.
You would need to include the full error message (which could be more
than one line) and the java stacktrace (which will be MANY lines). This
may mean you need to find the logfile, not something that shows up in
the admin UI. We'd also need to know the Solr version and as many other
details about your server, java, Solr startup, and index as you can
include. The config and schema would be great to have. Note that the
mailing list does not deal well with attachments, so anything like
screenshots and configs will generally need to be hosted elsewhere and
linked.
> So what is the solution that you can suggest me to use? Active Messagin
> (ActiveMQ or RabbitMQ) maybe?
>
> *PS: *
> -- By adding a *sleep() *in each of these scripts, the index managed to not
> be corrupted, that's why I'm suspecting that the number of concurrent
> request (*update*) received by solr, are causing this error.
Solr is fully threadsafe. It can handle multiple concurrent requests.
There was a relatively recent problem with multiple concurrent indexing
requests against Solr when running in SolrCloud mode with many shards,
but as far as I know this did not lead to problems starting a searcher,
it caused indexing to hang and not proceed.
Thanks,
Shawn