You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-commits@lucene.apache.org by yo...@apache.org on 2009/10/09 23:45:49 UTC
svn commit: r823705 - /lucene/solr/trunk/CHANGES.txt
Author: yonik
Date: Fri Oct 9 21:45:49 2009
New Revision: 823705
URL: http://svn.apache.org/viewvc?rev=823705&view=rev
Log:
doc - per-segment fieldcache issues
Modified:
lucene/solr/trunk/CHANGES.txt
Modified: lucene/solr/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/solr/trunk/CHANGES.txt?rev=823705&r1=823704&r2=823705&view=diff
==============================================================================
--- lucene/solr/trunk/CHANGES.txt (original)
+++ lucene/solr/trunk/CHANGES.txt Fri Oct 9 21:45:49 2009
@@ -30,6 +30,20 @@
faster for most cases. One can revert to the previous algorithm (which has
also been improved somewhat) by adding facet.method=enum to the request.
+Searching and sorting is now done on a per-segment basis, meaning that
+the FieldCache entries used for sorting and for function queries are
+created and used per-segment and can be reused for segments that don't
+change between index updates. While generally beneficial, this can lead
+to increased memory usage over 1.3 in certain scenarios:
+ 1) A single valued field that was used for both sorting and faceting
+in 1.3 would have used the same top level FieldCache entry. In 1.4,
+sorting will use entries at the segment level while faceting will still
+use entries at the top reader level, leading to increased memory usage.
+ 2) Certain function queries such as ord() and rord() require a top level
+FieldCache instance and can thus lead to increased memory usage. Consider
+replacing ord() and rord() with alternatives, such as function queries
+based on ms() for date boosting.
+
If you use custom Tokenizer or TokenFilter components in a chain specified in
schema.xml, they must support reusability. If your Tokenizer or TokenFilter
maintains state, it should implement reset(). If your TokenFilteFactory does
Re: svn commit: r823705 - /lucene/solr/trunk/CHANGES.txt
Posted by Yonik Seeley <yo...@lucidimagination.com>.
I don't envy casual users trying to make sense out of some of these
release notes... any suggestions on how to make it easier are welcome.
-Yonik
http://www.lucidimagination.com
On Fri, Oct 9, 2009 at 5:45 PM, <yo...@apache.org> wrote:
> Author: yonik
> Date: Fri Oct 9 21:45:49 2009
> New Revision: 823705
>
> URL: http://svn.apache.org/viewvc?rev=823705&view=rev
> Log:
> doc - per-segment fieldcache issues
>
> Modified:
> lucene/solr/trunk/CHANGES.txt
>
> Modified: lucene/solr/trunk/CHANGES.txt
> URL: http://svn.apache.org/viewvc/lucene/solr/trunk/CHANGES.txt?rev=823705&r1=823704&r2=823705&view=diff
> ==============================================================================
> --- lucene/solr/trunk/CHANGES.txt (original)
> +++ lucene/solr/trunk/CHANGES.txt Fri Oct 9 21:45:49 2009
> @@ -30,6 +30,20 @@
> faster for most cases. One can revert to the previous algorithm (which has
> also been improved somewhat) by adding facet.method=enum to the request.
>
> +Searching and sorting is now done on a per-segment basis, meaning that
> +the FieldCache entries used for sorting and for function queries are
> +created and used per-segment and can be reused for segments that don't
> +change between index updates. While generally beneficial, this can lead
> +to increased memory usage over 1.3 in certain scenarios:
> + 1) A single valued field that was used for both sorting and faceting
> +in 1.3 would have used the same top level FieldCache entry. In 1.4,
> +sorting will use entries at the segment level while faceting will still
> +use entries at the top reader level, leading to increased memory usage.
> + 2) Certain function queries such as ord() and rord() require a top level
> +FieldCache instance and can thus lead to increased memory usage. Consider
> +replacing ord() and rord() with alternatives, such as function queries
> +based on ms() for date boosting.
> +
> If you use custom Tokenizer or TokenFilter components in a chain specified in
> schema.xml, they must support reusability. If your Tokenizer or TokenFilter
> maintains state, it should implement reset(). If your TokenFilteFactory does
>
>
>