You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Russell Black (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 02:33:57 UTC
[jira] [Created] (SOLR-3222) Pull optimal cache warming queries
from a warm solr instance
Pull optimal cache warming queries from a warm solr instance
------------------------------------------------------------
Key: SOLR-3222
URL: https://issues.apache.org/jira/browse/SOLR-3222
Project: Solr
Issue Type: New Feature
Components: search
Affects Versions: 3.5, 4.0
Reporter: Russell Black
Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
It can even be used in a dynamic faction like this:
{code:xml}
<listener event="firstSearcher" class="solr.QuerySenderListener">
<xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
</listener>
{code}
which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
I implemented this by introducing a new request handler:
{code:xml}
<requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
{code}
The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
Patch to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3222) Pull optimal cache warming queries
from a warm solr instance
Posted by "Russell Black (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226179#comment-13226179 ]
Russell Black commented on SOLR-3222:
-------------------------------------
It is possible to configure how many entries to return from each of the caches. The default is 100 from each of the three caches. To change this, you can pass any of the following parameters to the request handler:
http://host/solr/core/autowarm?filterCache=50&fieldValueCache=150&queryResultCache=200
These parameters could also be specified as defaults in solrconfig.xml:
{code:xml}
<requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
<lst name="defaults">
<str name="filterCache">50</str>
<str name="fieldValueCache">150</str>
<str name="queryResultCache">200</str>
</lst>
</requestHandler>
{code}
> Pull optimal cache warming queries from a warm solr instance
> ------------------------------------------------------------
>
> Key: SOLR-3222
> URL: https://issues.apache.org/jira/browse/SOLR-3222
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 3.5, 4.0
> Reporter: Russell Black
> Labels: patch, performance
> Attachments: SOLR-3222-autowarm.patch
>
>
> Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
> One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
> It can even be used in a dynamic fashion like this:
> {code:xml}
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
> </listener>
> {code}
> which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
> I implemented this by introducing a new request handler:
> {code:xml}
> <requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
> {code}
> The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
> Patch to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3222) Pull optimal cache warming queries
from a warm solr instance
Posted by "Russell Black (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Black updated SOLR-3222:
--------------------------------
Description:
Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
It can even be used in a dynamic fashion like this:
{code:xml}
<listener event="firstSearcher" class="solr.QuerySenderListener">
<xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
</listener>
{code}
which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
I implemented this by introducing a new request handler:
{code:xml}
<requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
{code}
The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
Patch to follow.
was:
Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
It can even be used in a dynamic faction like this:
{code:xml}
<listener event="firstSearcher" class="solr.QuerySenderListener">
<xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
</listener>
{code}
which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
I implemented this by introducing a new request handler:
{code:xml}
<requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
{code}
The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
Patch to follow.
> Pull optimal cache warming queries from a warm solr instance
> ------------------------------------------------------------
>
> Key: SOLR-3222
> URL: https://issues.apache.org/jira/browse/SOLR-3222
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 3.5, 4.0
> Reporter: Russell Black
> Labels: patch, performance
>
> Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
> One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
> It can even be used in a dynamic fashion like this:
> {code:xml}
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
> </listener>
> {code}
> which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
> I implemented this by introducing a new request handler:
> {code:xml}
> <requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
> {code}
> The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
> Patch to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3222) Pull optimal cache warming queries
from a warm solr instance
Posted by "Russell Black (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Black updated SOLR-3222:
--------------------------------
Attachment: SOLR-3222-autowarm.patch
> Pull optimal cache warming queries from a warm solr instance
> ------------------------------------------------------------
>
> Key: SOLR-3222
> URL: https://issues.apache.org/jira/browse/SOLR-3222
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 3.5, 4.0
> Reporter: Russell Black
> Labels: patch, performance
> Attachments: SOLR-3222-autowarm.patch
>
>
> Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
> One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
> It can even be used in a dynamic fashion like this:
> {code:xml}
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
> </listener>
> {code}
> which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
> I implemented this by introducing a new request handler:
> {code:xml}
> <requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
> {code}
> The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
> Patch to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Closed] (SOLR-3222) Pull optimal cache warming queries from
a warm solr instance
Posted by "Russell Black (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Black closed SOLR-3222.
-------------------------------
Resolution: Incomplete
Turns out this patch doesn't work, since there is no reliable way to turn a Query object into URL query parameters. I ended up solving the problem with a cache plugin. Let me know if you're interested in the solution and I can post code.
> Pull optimal cache warming queries from a warm solr instance
> ------------------------------------------------------------
>
> Key: SOLR-3222
> URL: https://issues.apache.org/jira/browse/SOLR-3222
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 3.5, 4.0
> Reporter: Russell Black
> Labels: patch, performance
> Attachments: SOLR-3222-autowarm.patch
>
>
> Ever wondered what queries to use to prime your cache? This patch allows you to query a warm running instance for a list of warming queries. The list is generated from the server's caches, meaning you get back an optimal set of queries. The set is optimal to the extent that the caches are optimized. The queries are returned in a format that can be consumed by the {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code} section of {{solrconfig.xml}}.
> One can use this feature to generate a static set of good warming queries to place in {{solrconfig.xml}} under {code:xml}<listener event="firstSearcher" class="solr.QuerySenderListener">{code}
> It can even be used in a dynamic fashion like this:
> {code:xml}
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <xi:include href="http://host/solr/core/autowarm" xpointer="element(/1/2)" xmlns:xi="http://www.w3.org/2001/XInclude"/>
> </listener>
> {code}
> which can work well in certain distributed load-balanced architectures, although in production it would be wise to add an {{<xi:fallback>}} element to the include in the event that the host is down.
> I implemented this by introducing a new request handler:
> {code:xml}
> <requestHandler name="/autowarm" class="solr.AutoWarmRequestHandler" />
> {code}
> The request handler pulls a configurable number of "top" keys from the {{filterCache}},{{fieldValueCache}}, and {{queryResultCache}}. For each key, it constructs a query that will cause that key to be placed in the associated cache. The list of constructed queries are then returned in the response.
> Patch to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org