You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by steven crichton <st...@mac.com> on 2013/12/04 20:30:58 UTC
Solr Stalls on Bulk indexing, no logs or errors
I am finding with a bulk index using SOLR 4.3 on Tomcat, that when I reach
69578 records the server stops adding anything more.
I've tried reducing the data sent to the bare minimum of fields and using
ASC and DESC data to see if it could be a field issue.
Is there anything I could look at for this? As I'm not finding anything
similar noted before. Does tomcat have issues with closing connections that
look like DDOS attacks? Or could it be related to too many commits in too
short a time?
Any help will be very greatly appreciated.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Stalls on Bulk indexing, no logs or errors
Posted by Erick Erickson <er...@gmail.com>.
Wait, crashes? Or just stops accepting updates?
At any rate, this should be fixed in 4.6. If you
can dump a stack trace, we can identify whether this
is the same issue quickly. jstack is popular.
If you're still having queries served, it's probably
not your commit settings, try searching the
JIRA list for "distributed deadlock". You should
find two JIRAs, one relevant to SolrJ by Joel
Bernstein (probably not one you are about) and
one by Mark Miller that address this.
Best,
Erick
On Wed, Dec 4, 2013 at 3:19 PM, steven crichton <st...@mac.com>wrote:
> Yes I can continue to query after this importer goes down and whilst it
> running.
>
> The bulk commit is done via a JSON handler in php. There is 121,000
> records that need to go into the index. So this is done in 5000 chunked
> mySQL retrieve calls and parsing to the data as required.
>
> workflow:
>
> get record
> create {add doc… } JSON
> Post to CORE/update/json
>
>
> I stopped doing a hard commit every 1000 records. To see if that was an
> issue.
>
>
> the auto commit settings are ::
>
> <autoCommit>
> <maxDocs>${solr.autoCommit.MaxDocs:5000}</maxDocs>
> <maxTime>${solr.autoCommit.MaxTime:24000}</maxTime>
> </autoCommit>
>
>
> I’ve pretty much worked out of the drupal schemas for SOLR 4
> https://drupal.org/project/apachesolr
>
> At one point I thought it could be malformed data, but even reducing the
> records down to just the id and title now .. it crashes at the same point.
> As in the query still works but the import handler does nothing at all
>
>
> Tomcat logs seem to indicate no major issues.
>
>
> There’s not a strange variable that is set to make an upper index limit is
> there?
>
> Regards,
> Steven
>
>
>
> On 4 Dec 2013, at 20:02, Erick Erickson [via Lucene] <
> ml-node+s472066n4104984h38@n3.nabble.com> wrote:
>
> > There's a known issue with SolrCloud with multiple shards, but
> > you haven't told us whether you're using that. The test for
> > whether you're running in to that is whether you can continue
> > to _query_, just not update.
> >
> > But you need to tell us more about our setup. In particular
> > hour commit settings (hard and soft), your solrconfig settings,
> > particularly around autowarming, how you're "bulk indexing",
> > SolrJ? DIH? a huge CSV file?
> >
> > Best,
> > Erick
> >
> >
> > On Wed, Dec 4, 2013 at 2:30 PM, steven crichton <[hidden email]>wrote:
> >
> > > I am finding with a bulk index using SOLR 4.3 on Tomcat, that when I
> reach
> > > 69578 records the server stops adding anything more.
> > >
> > > I've tried reducing the data sent to the bare minimum of fields and
> using
> > > ASC and DESC data to see if it could be a field issue.
> > >
> > > Is there anything I could look at for this? As I'm not finding anything
> > > similar noted before. Does tomcat have issues with closing connections
> that
> > > look like DDOS attacks? Or could it be related to too many commits in
> too
> > > short a time?
> > >
> > > Any help will be very greatly appreciated.
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> >
> >
> > If you reply to this email, your message will be added to the discussion
> below:
> >
> http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981p4104984.html
> > To unsubscribe from Solr Stalls on Bulk indexing, no logs or errors,
> click here.
> > NAML
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981p4104990.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr Stalls on Bulk indexing, no logs or errors
Posted by steven crichton <st...@mac.com>.
Yes I can continue to query after this importer goes down and whilst it running.
The bulk commit is done via a JSON handler in php. There is 121,000 records that need to go into the index. So this is done in 5000 chunked mySQL retrieve calls and parsing to the data as required.
workflow:
get record
create {add doc… } JSON
Post to CORE/update/json
I stopped doing a hard commit every 1000 records. To see if that was an issue.
the auto commit settings are ::
<autoCommit>
<maxDocs>${solr.autoCommit.MaxDocs:5000}</maxDocs>
<maxTime>${solr.autoCommit.MaxTime:24000}</maxTime>
</autoCommit>
I’ve pretty much worked out of the drupal schemas for SOLR 4
https://drupal.org/project/apachesolr
At one point I thought it could be malformed data, but even reducing the records down to just the id and title now .. it crashes at the same point. As in the query still works but the import handler does nothing at all
Tomcat logs seem to indicate no major issues.
There’s not a strange variable that is set to make an upper index limit is there?
Regards,
Steven
On 4 Dec 2013, at 20:02, Erick Erickson [via Lucene] <ml...@n3.nabble.com> wrote:
> There's a known issue with SolrCloud with multiple shards, but
> you haven't told us whether you're using that. The test for
> whether you're running in to that is whether you can continue
> to _query_, just not update.
>
> But you need to tell us more about our setup. In particular
> hour commit settings (hard and soft), your solrconfig settings,
> particularly around autowarming, how you're "bulk indexing",
> SolrJ? DIH? a huge CSV file?
>
> Best,
> Erick
>
>
> On Wed, Dec 4, 2013 at 2:30 PM, steven crichton <[hidden email]>wrote:
>
> > I am finding with a bulk index using SOLR 4.3 on Tomcat, that when I reach
> > 69578 records the server stops adding anything more.
> >
> > I've tried reducing the data sent to the bare minimum of fields and using
> > ASC and DESC data to see if it could be a field issue.
> >
> > Is there anything I could look at for this? As I'm not finding anything
> > similar noted before. Does tomcat have issues with closing connections that
> > look like DDOS attacks? Or could it be related to too many commits in too
> > short a time?
> >
> > Any help will be very greatly appreciated.
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981p4104984.html
> To unsubscribe from Solr Stalls on Bulk indexing, no logs or errors, click here.
> NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981p4104990.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Stalls on Bulk indexing, no logs or errors
Posted by Erick Erickson <er...@gmail.com>.
There's a known issue with SolrCloud with multiple shards, but
you haven't told us whether you're using that. The test for
whether you're running in to that is whether you can continue
to _query_, just not update.
But you need to tell us more about our setup. In particular
hour commit settings (hard and soft), your solrconfig settings,
particularly around autowarming, how you're "bulk indexing",
SolrJ? DIH? a huge CSV file?
Best,
Erick
On Wed, Dec 4, 2013 at 2:30 PM, steven crichton <st...@mac.com>wrote:
> I am finding with a bulk index using SOLR 4.3 on Tomcat, that when I reach
> 69578 records the server stops adding anything more.
>
> I've tried reducing the data sent to the bare minimum of fields and using
> ASC and DESC data to see if it could be a field issue.
>
> Is there anything I could look at for this? As I'm not finding anything
> similar noted before. Does tomcat have issues with closing connections that
> look like DDOS attacks? Or could it be related to too many commits in too
> short a time?
>
> Any help will be very greatly appreciated.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-Stalls-on-Bulk-indexing-no-logs-or-errors-tp4104981.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>