You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Martijn van Groningen (JIRA)" <ji...@apache.org> on 2011/08/20 13:56:27 UTC
[jira] [Created] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
TieredMergePolicy and expungeDeletes behaviour
----------------------------------------------
Key: SOLR-2725
URL: https://issues.apache.org/jira/browse/SOLR-2725
Project: Solr
Issue Type: Bug
Affects Versions: 3.3
Reporter: Martijn van Groningen
Fix For: 3.4, 4.0
During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
My solconfig.xml (that uses TieredMergePolicy, b/c luceneMatchVersion=3.3) contained:
{code:xml}
<indexDefaults>
<mergeFactor>5</mergeFactor>
{code}
However there were still ~30 segments left with deletes after the commit finished.
After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setMaxMergeAtOnceExplicit and
TieredMergePolicy#setExpungeDeletesPctAllowed aren't invoked.
I think the following statements should be added to the SolrIndexConfig#buildMergePolicy method:
{code}
tieredMergePolicy.setMaxMergeAtOnceExplicit(mergeFactor);
tieredMergePolicy.setExpungeDeletesPctAllowed(0);
{code}
I think these changes should reflect the behavior of Solr 3.1 / 3.2
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martijn van Groningen updated SOLR-2725:
----------------------------------------
Affects Version/s: 3.6
3.4
3.5
Fix Version/s: (was: 3.6)
Removed 3.6 from fix versions.
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3, 3.4, 3.5, 3.6
> Reporter: Martijn van Groningen
> Fix For: 4.0
>
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Martijn van Groningen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martijn van Groningen updated SOLR-2725:
----------------------------------------
Description:
During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
However there were still ~30 segments left with deletes after the commit finished.
After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
{code}
tieredMergePolicy.setExpungeDeletesPctAllowed(0);
{code}
This also reflects the behavior of Solr 3.1 / 3.2
After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
{code:xml}
<expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
{code}
Also having a expungeDeletesPctAllowed per commit command would be great:
{code:xml}
<commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
{code}
was:
During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
My solconfig.xml (that uses TieredMergePolicy, b/c luceneMatchVersion=3.3) contained:
{code:xml}
<indexDefaults>
<mergeFactor>5</mergeFactor>
{code}
However there were still ~30 segments left with deletes after the commit finished.
After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setMaxMergeAtOnceExplicit and
TieredMergePolicy#setExpungeDeletesPctAllowed aren't invoked.
I think the following statements should be added to the SolrIndexConfig#buildMergePolicy method:
{code}
tieredMergePolicy.setMaxMergeAtOnceExplicit(mergeFactor);
tieredMergePolicy.setExpungeDeletesPctAllowed(0);
{code}
I think these changes should reflect the behavior of Solr 3.1 / 3.2
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3
> Reporter: Martijn van Groningen
> Fix For: 3.4, 4.0
>
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Shawn Heisey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507838#comment-13507838 ]
Shawn Heisey commented on SOLR-2725:
------------------------------------
It occurred to me that in addition to allowing solrconfig.xml to set this value, that it should also be available via SolrJ. Which brings up SOLR-1487.
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3, 3.4, 3.5, 3.6
> Reporter: Martijn van Groningen
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Martijn van Groningen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martijn van Groningen updated SOLR-2725:
----------------------------------------
Fix Version/s: (was: 3.4)
3.5
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3
> Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Martijn van Groningen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martijn van Groningen updated SOLR-2725:
----------------------------------------
Fix Version/s: (was: 4.0)
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3, 3.4, 3.5, 3.6
> Reporter: Martijn van Groningen
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2725) TieredMergePolicy and expungeDeletes
behaviour
Posted by "Shawn Heisey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475182#comment-13475182 ]
Shawn Heisey commented on SOLR-2725:
------------------------------------
This is probably an unnecessary comment, but I wanted to be sure that when this is implemented, it's done as a double/float so that values significantly less than 1% are possible. It looks like the underlying mechanism is already set up this way:
private double forceMergeDeletesPctAllowed = 10.0;
> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
> Key: SOLR-2725
> URL: https://issues.apache.org/jira/browse/SOLR-2725
> Project: Solr
> Issue Type: Bug
> Affects Versions: 3.3, 3.4, 3.5, 3.6
> Reporter: Martijn van Groningen
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code}
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org