[CONF] Apache Solr Reference Guide > Major Changes from Solr 3 to Solr 4

["Hoss Man (Confluence)" <co...@apache.org>]

Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr)
Page: Major Changes from Solr 3 to Solr 4 (https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+3+to+Solr+4)

Change Comment:
---------------------------------------------------------------------
No-Op edit of a long page to tests email notification behavior

Edited by Hoss Man:
---------------------------------------------------------------------
Solr 4 includes some exciting new developments, and also includes many changes from Solr 3.x and earlier.

{toc}

h2. Highlights of Solr 4

Solr 4 is a major release of Solr, two years in the making, and includes new features for scalability and high performance for today's data driven, real time search applications. Some of the major improvements include:

*SolrCloud*

The primary new feature in Solr 4 goes by the name "SolrCloud", a suite of tools to make scalability built into your project from day one:

* Distributed indexing designed from the ground up for near real-time (NRT) and NoSQL features such as realtime-get, optimistic locking, and durable updates.
* High availability with no single points of failure.
* Apache Zookeeper integration for distributed coordination and cluster metadata and configuration storage.
* Immunity to split-brain issues due to Zookeeper's Paxos distributed consensus protocols.
* Updates sent to any node in the cluster and are automatically forwarded to the correct shard and replicated to multiple nodes for redundancy.
* Queries sent to any node automatically perform a full distributed search across the cluster with load balancing and fail-over.

*NoSQL Features*

Users wishing to use Solr as their primary data store will be interested in these features:

* Update durability - A transaction log ensures that even uncommitted documents are never lost.
* Real-time Get - The ability to quickly retrieve the latest version of a document, without the need to commit or open a new searcher
* Versioning and Optimistic Locking - combined with real-time get, this allows read-update-write functionality that ensures no conflicting changes were made concurrently by other clients.
* Atomic updates - the ability to add, remove, change, and increment fields of an existing document without having to send in the complete document again. 

*Other Major Features*

There's more: 

* Pivot Faceting - Multi-level or hierarchical faceting where the top constraints for one field are found for each top constraint of a different field.
* Pseudo-fields - The ability to alias fields, or to add metadata along with returned documents, such as function query values and results of spatial distance calculations.
* A spell checker implementation that can work directly from the main index instead of creating a sidecar index.
* Pseudo-Join functionality - The ability to select a set of documents based on their relationship to a second set of documents.
* Function query enhancements including conditional function queries and relevancy functions.
* New update processors to facilitate modifying documents prior to indexing.
* A brand new web admin interface, including support for SolrCloud.

h2. Changes to Consider

There are some major changes in Solr 4 to consider before starting to migrate your configurations and indexes. There are many hundreds of changes, so a thorough review of the changes.txt file in your Solr instance will help you plan migration to Solr 4.

h3. System Changes

* Java 1.6 is now required to run Solr 4.

h3. Index Format

* The Lucene index format has changed. As a result, once you upgrade to Solr 4, previous versions of Solr will no longer be able to read your indices. In a master/slave configuration, all searchers/slaves should be upgraded before the master. If the master is updated first, older searchers will not be able to read the new index format.

h3. Query Parsers

* The default logic for the {{mm}} parameter of the [Dismax Query Parser|solr:The DisMax Query Parser] has changed. If no {{mm}} parameter is specified (either in the query or as a default in {{solrconfig.xml}}, then the effective value of the {{q.op}} parameter is used to influence the behavior (whether {{q.op}} is defined in the query, in {{solrconfig.xml}}, or from the {{defaultOperator}} option in {{schema.xml}}). If {{q.op}} is effectively "AND" then {{mm=100%}}. If {{q.op}} is effectively "OR" then {{mm=0%}}. If you want to force legacy behavior, set a default value for the {{mm}} parameter in your {{solrconfig.xml}} file.

h3. Schema Configuration

* Due to low level changes to support SolrCloud, the {{uniqueKey field}} can no longer be poplated via {{<copyField/>}} or {{<field default=...>}} in {{schema.xml}}. If you want to have Solr automatically generate a uniqueKey value when adding documents, use an instance of solr.UUIDUpdateProcessorFactory in their update processor chain. See [SOLR-2798|https://issues.apache.org/jira/browse/SOLR-2796] for more details.

* Solr is now much more strict about requiring that the {{uniqueKeyField}} feature (if used) must refer to a field which is not multiValued. If you upgrade from an earlier version of Solr and see an error that your {{uniqueKeyField}} "can not be configured to be multivalued" please add {{multiValued="false}}' to the {{<field />}} declaration for your {{uniqueKeyField}}.

* Changes to the {{HTMLCharFilterFactory}}:
** Known offset bugs have been fixed.
** The "Mark invalid" exceptions are no longer triggered.
** Newlines are now substituted instead of spaces for block-level elements; this corresponds more closely to on-screen layout, enables sentence segmentation, and doesn't change the offsets.
** Supplementary characters in tags are now recognized.
** Accepted tag names have been switched from {{\[:XID_Start:\]}} and {{\[:XID_Continue:\]}} Unicode properties to the more relaxed {{\[:ID_Start:\]}} and {{\[:ID_Continue:\]}} properties, in order to broaden the range of recognizable input.  (The improved security afforded by the {{XID_\*}} properties is irrelevant to what a {{CharFilter}} does.)
** More cases of <script> tags are now properly stripped.
** CDATA sections are now recognized.&nbsp;
** No space is substituted for inline tags (e.g. {{<b>}}, {{<i>}}, {{<span>}}).  The old version substituted spaces for all tags.
** Broken MS-Word-generated processing instructions {{(? ... /)}} instead of {{<? ... ?>}}) are now handled.
** Uppercase character entities "", "©", "<", ">", "®", and  "&" are now recognized and handled as if they were lower case.
** Opening tags with unbalanced quotation marks are now properly stripped.
** Literal "<" and ">" characters in opening tags, regardless of whether they appear inside quotation marks, now inhibit recognition (and stripping) of the tags.  The only exception to this is for values of event-handler attributes, e.g. "onClick", "onLoad", "onSelect".
** A newline '\n' is substituted instead of a space for stripped HTML markup.
** Nothing is substituted for opening and closing inline tags - they are simply removed. The list of inline tags is (case insensitively): <a>, <abbr>, <acronym>, <b>, <basefont>, <bdo>, <big>, <cite>, <code>, <dfn>, <em>, <font>, <i>, <img>, <input>, <kbd>, <label>, <q>, <s>, <samp>, <select>, <small>, <span>, <strike>,   <strong>, <sub>, <sup>, <textarea>, <tt>, <u>, and <var>.
** HTMLStripCharFilterFactory now handles HTMLStripCharFilter's "escapedTags" feature: opening and closing tags with the given names, including any attributes and their values, are left intact in the output.
** The replacement character U+FFFD is now used to replace numeric character entities for unpaired UTF-16 low and high surrogates (in the range \[U+D800-U+DFFF\]).
** Properly paired numeric character entities for UTF-16 surrogates are now converted to the corresponding code units.
** The generated scanner's parse method has been changed from the default {{yylex()}} to {{nextChar()}}.

h3. Changes to solrconfig.xml

* The <indexDefaults> and <mainIndex> sections of solrconfig.xml have been discontinued and replaced with the <indexConfig> section. There are also better defaults. When migrating, If you don't know what your old settings mean, delete both the <indexDefaults> and <mainIndex> sections. If you have customized them, put them in the <indexConfig> section with the same syntax as before.

* The {{PingRequestHandler}} no longer looks for a {{<healthcheck>}} option in the (legacy) {{<admin>}} section of {{solrconfig.xml}}. If you want to take advantage of this feature, configure a {{healthcheckFile}} initialization parameter directly on the {{PingRequestHandler}}. As part of this change, relative file paths have been fixed to be resolved against the data directory. The sample {{solrconfig.xml}} has an example of this configuration.

* The update request parameter to choose the Update Request Processor Chain has been renamed from {{update.processor}} to {{update.chain}}. The old parameter was deprecated in Solr 3.x, but now has been removed entirely.

* The VelocityResponseWriter is no longer built into the core. Its jar and dependencies now need to be addressed (via <lib> or solr/home lib inclusion). It also needs to be registered in {{solrconfig.xml}} like this:\\
{code:lang=xml|borderStyle=solid|borderColor=#666666}
<queryResponseWritername="velocity" class="solr.VelocityResponseWriter"/>
{code}

h3. Other Changes

* Two of the SolrServer subclasses in SolrJ have been renamed and replaced. {{CommonsHttpSolrServer}} is now {{HttpSolrServer}}, and {{StreamingUpdateSolrServer}} is now {{ConcurrentUpdateSolrServer}}.

{scrollbar}




Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr
Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action