You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rudolf Grigeľ <gr...@gmail.com> on 2015/06/23 08:23:31 UTC
DIH deletes cause opening of searchers
Hello,
I recently found out weird behavior while using the DIH with the Solr 5.2.
I am using extended line entity processor which can also delete documents
based on some special conditions (delete by query).
The problem is that the DIH is incredibly slow compared to Solr 4.3. I have
noticed in the logs that after every delete statement
a new searcher is opened. I checked the solrconfig.xml and autoCommit is
set for 15 seconds and openSearcher is set to false, softAutoCommit section
is commented out. I have also tried to comment out autoCommit section but
the behavior is the same.
This is the relevant part of the log:
INFO - 2015-06-23 06:07:33.510; [ test_core]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: (KNUMH:"0084655650")AND(KSCHL:"ZKA0")
INFO - 2015-06-23 06:07:33.519; [ test_core]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@60060afc[test_core]
realtime
INFO - 2015-06-23 06:07:33.520; [ test_core]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: (KNUMH:"0084655651")AND(KSCHL:"ZKA0")
INFO - 2015-06-23 06:07:33.523; [ test_core]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@73d2a715[test_core]
realtime
INFO - 2015-06-23 06:07:33.523; [ test_core]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: (KNUMH:"0084655652")AND(KSCHL:"ZKA0")
INFO - 2015-06-23 06:07:33.534; [ test_core]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@b52e1e9[test_core]
realtime
INFO - 2015-06-23 06:07:33.534; [ test_core]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: (KNUMH:"0084655653")AND(KSCHL:"ZKA0")
INFO - 2015-06-23 06:07:33.537; [ test_core]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@d8f211f[test_core]
realtime
INFO - 2015-06-23 06:07:33.537; [ test_core]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: (KNUMH:"0084655654")AND(KSCHL:"ZKA0")
Is this behavior normal ?? How can I prevent opening new searcher after
every delete statement ? It is sufficient for me when the changes are
visible when the DIH finishes its job.
Thank you
--
Rudolf Grigeľ
Re: DIH deletes cause opening of searchers
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Tue, Jun 30, 2015 at 7:17 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 6/25/2015 2:20 AM, Mikhail Khludnev wrote:
>
>> On Tue, Jun 23, 2015 at 9:23 AM, Rudolf Grigeľ <gr...@gmail.com>
>> wrote:
>>
>>> How can I prevent opening new searcher after
>>> every delete statement ?
>>>
>> comment <updateLog> tag in solrconfig.xml (it always help)
>>
>
> The presence or absence of the updateLog should not affect whether new
> searchers are opened.
Note, it's a realtime searcher, not the plain one.
I reproduced it with the recent release, overall movie seems too long and
boring, here is the crux moment
https://youtu.be/S941iqK9hpw?t=755
here are logs excerpts:
by default <updateLog>
INFO - 2015-07-01 06:57:05.257; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:F8
V7067-APL-KIT
INFO - 2015-07-01 06:57:05.363; [ db]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@f5234ae[db]
realtime
INFO - 2015-07-01 06:57:05.385; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:IW
-02
INFO - 2015-07-01 06:57:05.438; [ db]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@287b075b[db]
realtime
INFO - 2015-07-01 06:57:05.468; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:MA
147LL/A
INFO - 2015-07-01 06:57:05.482; [ db]
org.apache.solr.search.SolrIndexSearcher; Opening Searcher@4483483f[db]
realtime
INFO - 2015-07-01 06:57:05.491; [ db]
org.apache.solr.handler.dataimport.DocBuilder; Import completed successfully
after <!-- updateLog
INFO - 2015-07-01 07:04:14.533; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:F8
V7067-APL-KIT
INFO - 2015-07-01 07:04:14.541; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:IW
-02
INFO - 2015-07-01 07:04:14.553; [ db]
org.apache.solr.handler.dataimport.SolrWriter; Deleting documents from Solr
with query: id:MA
147LL/A
> If this change actually works, I'm pretty sure that's a bug.
>
> IMHO the updateLog should always be enabled on Solr 4.x and up, and
> autoCommit with openSearcher set to false should be configured so the
> transaction logs do not get huge.
>
> Thanks,
> Shawn
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: DIH deletes cause opening of searchers
Posted by Erick Erickson <er...@gmail.com>.
>From the log fragment it's at least worth further investigation.
You've had 4 searchers open in less than 1/2 second. That's
horribly fast, but you already know that...
Let's see the DIH configs, perhaps there's something
innocent-seeming there that's causing this. Or, there's
a bug somewhere.
Best,
Erick
On Tue, Jun 30, 2015 at 12:17 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 6/25/2015 2:20 AM, Mikhail Khludnev wrote:
>>
>> On Tue, Jun 23, 2015 at 9:23 AM, Rudolf Grigeľ <gr...@gmail.com> wrote:
>>>
>>> How can I prevent opening new searcher after
>>> every delete statement ?
>>
>> comment <updateLog> tag in solrconfig.xml (it always help)
>
>
> The presence or absence of the updateLog should not affect whether new
> searchers are opened. If this change actually works, I'm pretty sure that's
> a bug.
>
> IMHO the updateLog should always be enabled on Solr 4.x and up, and
> autoCommit with openSearcher set to false should be configured so the
> transaction logs do not get huge.
>
> Thanks,
> Shawn
>
Re: DIH deletes cause opening of searchers
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/25/2015 2:20 AM, Mikhail Khludnev wrote:
> On Tue, Jun 23, 2015 at 9:23 AM, Rudolf Grigeľ <gr...@gmail.com> wrote:
>> How can I prevent opening new searcher after
>> every delete statement ?
> comment <updateLog> tag in solrconfig.xml (it always help)
The presence or absence of the updateLog should not affect whether new
searchers are opened. If this change actually works, I'm pretty sure
that's a bug.
IMHO the updateLog should always be enabled on Solr 4.x and up, and
autoCommit with openSearcher set to false should be configured so the
transaction logs do not get huge.
Thanks,
Shawn
Re: DIH deletes cause opening of searchers
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Tue, Jun 23, 2015 at 9:23 AM, Rudolf Grigeľ <gr...@gmail.com> wrote:
> How can I prevent opening new searcher after
> every delete statement ?
>
comment <updateLog> tag in solrconfig.xml (it always help)
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>