You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Jörn Franke (Jira)" <ji...@apache.org> on 2020/01/31 20:36:00 UTC
[jira] [Comment Edited] (SOLR-14202) DIH with a ScriptTransformer
that does Atomic Updates leaks searchers
[ https://issues.apache.org/jira/browse/SOLR-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027810#comment-17027810 ]
Jörn Franke edited comment on SOLR-14202 at 1/31/20 8:35 PM:
-------------------------------------------------------------
I added the zip file eoedihleak.zip
It contains a conf folder => this is a solr conf with a DIH /dataimport/searcherLeak
Upload it Solr and create a collection based on that
place the file importfile/leaksampleimport.xml into /tmp/eoe on the Solr server (make sure Solr can access it). If you want to put it in another folder then you have to edit conf/dataConfigs/searchLeakImport.xml and change it there.
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher (hit Refresh Values before). You will see exactly one Searcher open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit)
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher (hit Refresh Values before). You will see exactly one Searcher open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit) again
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher (hit Refresh Values before). You will see exactly two Searchers open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit) again
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher (hit Refresh Values before). You will see exactly three Searchers open
...
and so on. This opened searchers will cause that the outdated index files are not deleted until you stop solr and then during the next start of Solr the outdated index files are deleted.
I am pretty sure the issue is on 8.4 and 8.3, but possibly all 8.x version (not all tested). Maybe even before 8, but this I did not test.
was (Author: jornfranke):
I added the zip file eoedihleak.zip
It contains a conf folder => this is a solr conf with a DIH /dataimport/searcherLeak
Upload it Solr and create a collection based on that
place the file importfile/leaksampleimport.xml into /tmp/eoe on the Solr server (make sure Solr can access it). If you want to put it in another folder then you have to edit conf/dataConfigs/searchLeakImport.xml and change it there.
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher. You will see exactly one Searcher open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit)
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher. You will see exactly one Searcher open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit) again
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher. You will see exactly two Searchers open
Then run in Admin UI/collection/Dataimport the handler /dataimport/searchLeak (default with Commit) again
Monitor in Admin UI /core/Plugin/Stats/Core the number of searcher. You will see exactly three Searchers open
...
and so on. This opened searchers will cause that the outdated index files are not deleted until you stop solr and then during the next start of Solr the outdated index files are deleted.
I am pretty sure the issue is on 8.4 and 8.3, but possibly all 8.x version (not all tested). Maybe even before 8, but this I did not test.
> DIH with a ScriptTransformer that does Atomic Updates leaks searchers
> ---------------------------------------------------------------------
>
> Key: SOLR-14202
> URL: https://issues.apache.org/jira/browse/SOLR-14202
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 8.3, 8.4
> Reporter: Jörn Franke
> Priority: Major
> Attachments: eoe.zip, eoedihleak.zip
>
>
> The data directory of a collection is growing and growing. It seems that old segments are not deleted. They are only deleting during start of Solr.
> How to reproduce. Have any collection (e.g. the example collection) and start indexing documents. Even during the indexing the data directory is growing significantly - much more than expected (several magnitudes). if certain documents are updated (without significantly increasing the amount of data) the index data directory grows again several magnitudes. Even for small collections the needed space explodes.
> This reduces significantly if Solr is stopped and then started. During startup (not shutdown) Solr purges all those segments if not needed (* sometimes some but not a significant amount is deleted during shutdown). This is of course not a good workaround for normal operations.
> It does not seem to have a affect on queries (their performance do not seem to change).
> The configs have not changed before the upgrade and after (e.g. from Solr 8.2 to 8.3 to 8.4, not cross major versions), so I assume it could be related to Solr 8.4. It may have been also in Solr 8.3 (not sure), but not in 8.2.
>
> IndexConfig is pretty much default: Lock type: native, autoCommit: 15000, openSearcher=false, autoSoftCommit -1 (reproducible with autoCommit 5000).
> Nevertheless, it did not happen in previous versions of Solr and the config did not change.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org