You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Candygram For Mongo <ca...@gmail.com> on 2014/04/04 02:26:33 UTC
Full Indexing is Causing a Java Heap Out of Memory Exception
*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
*SOLR/Lucene version: *4.2.1*
*JVM version:
Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
*Indexer startup command:
set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
java " %JVMARGS% ^
-Dcom.sun.management.jmxremote.port=1092 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false ^
-jar start.jar
*SOLR indexing HTTP parameters request:
webapp=/solr path=/dataimport
params={clean=false&command=full-import&wt=javabin&version=2}
We are getting a Java heap OOM exception when indexing (updating) 27
million records. If we increase the Java heap memory settings the problem
goes away but we believe the problem has not been fixed and that we will
eventually get the same OOM exception. We have other processes on the
server that also require resources so we cannot continually increase the
memory settings to resolve the OOM issue. We are trying to find a way to
configure the SOLR instance to reduce or preferably eliminate the
possibility of an OOM exception.
We can reproduce the problem on a test machine. We set the Java heap
memory size to 64MB to accelerate the exception. If we increase this
setting the same problems occurs, just hours later. In the test
environment, we are using the following parameters:
JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
Normally we use the default solrconfig.xml file with only the following jar
file references added:
<lib path="../../../../default/lib/common.jar" />
<lib path="../../../../default/lib/webapp.jar" />
<lib path="../../../../default/lib/commons-pool-1.4.jar" />
Using these values and trying to index 6 million records from the database,
the Java Heap Out of Memory exception is thrown very quickly.
We were able to complete a successful indexing by further modifying the
solrconfig.xml and removing all or all but one <copyfield> tags from the
schema.xml file.
The following solrconfig.xml values were modified:
<ramBufferSizeMB>6</ramBufferSizeMB>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">2</int>
<int name="maxMergeAtOnceExplicit">2</int>
<int name="segmentsPerTier">10</int>
<int name="maxMergedSegmentMB">150</int>
</mergePolicy>
<autoCommit>
<maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
<openSearcher>false</openSearcher>
</autoCommit>
Using our customized schema.xml file with two or more <copyfield> tags, the
OOM exception is always thrown. Based on the errors, the problem occurs
when the process was trying to do the merge. The error is provided below:
Exception in thread "Lucene Merge Thread #156"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
at
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
at
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
at
org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
hit an OutOfMemoryError; cannot commit
at
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
at
org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
We think but are not 100% sure that the problem is related to the merge.
Normally our schema.xml contains a lot of field specifications (like the
ones seen in the file fragment below):
<copyField source="ADDRESS.RECORD_ID_abc" dest="ADDRESS.RECORD_ID.case_abc"
/>
<copyField source="ADDRESS.RECORD_ID_abc"
dest="ADDRESS.RECORD_ID.case.soundex_abc" />
<copyField source="ADDRESS.RECORD_ID_abc"
dest="ADDRESS.RECORD_ID.case_nvl_abc" />
In tests using the default file schema.xml and no <copyfield> tags,
indexing completed successfully. 6 million records produced a 900 MB data
directory.
When I included just one <copyfield> tag, indexing completed successfully. 6
million records produced a 990 MB data directory (90 MB bigger).
When I included just two <copyfield> tags, the index crashed with an OOM
exception.
Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
crash.
The net of our test results I as follows:
*solrconfig.xml*
*schema.xml*
*result*
default plus only jar references
default (no copyfield tags)
success
default plus only jar references
modified with one copyfield tag
success
default plus only jar references
modified with two copyfield tags
crash
additional modified settings
default (no copyfield tags)
success
additional modified settings
modified with one copyfield tag
success
additional modified settings
modified with two copyfield tags
crash
Our question is, what can we do to eliminate these OOM exceptions?
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
solrconfig.xml below
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
For more details about configurations options that may appear in
this file, see http://wiki.apache.org/solr/SolrConfigXml.
-->
<config>
<!-- For Database / JDBC / DIH / Data Import Hanlder -->
<!-- Solr 3.6: <lib dir="../../../dist/"
regex="apache-solr-dataimporthandler-.*\.jar" /> -->
<lib dir="../../../dist/" regex="solr-dataimporthandler-.*\.jar" />
<!-- In all configuration below, a prefix of "solr." for class names
is an alias that causes solr to search appropriate packages,
including org.apache.solr.(search|update|request|core|analysis)
You may also specify a fully qualified Java classname if you
have your own custom plugins.
-->
<abortOnConfigurationError>${solr.abortOnConfigurationError:true}</abortOnConfigurationError>
<!-- Controls what version of Lucene various components of Solr
adhere to. Generally, you want to use the latest version to
get all bug fixes and improvements. It is highly recommended
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
<luceneMatchVersion>LUCENE_42</luceneMatchVersion>
<!-- lib directives can be used to instruct Solr to load an Jars
identified and use them to resolve any "plugins" specified in
your solrconfig.xml or schema.xml (ie: Analyzers, Request
Handlers, etc...).
All directories and paths are resolved relative to the
instanceDir.
If a "./lib" directory exists in your instanceDir, all files
found in it are included as if you had used the following
syntax...
<lib dir="./lib" />
-->
<!-- A 'dir' option by itself adds any files found in the directory
to the classpath, this is useful for including all jars in a
directory.
-->
<!--
<lib dir="../add-everything-found-in-this-dir-to-the-classpath" />
-->
<!-- When a 'regex' is specified in addition to a 'dir', only the
files in that directory which completely match the regex
(anchored on both ends) will be included.
-->
<lib dir="../../../dist/" regex="apache-solr-cell-\d.*\.jar" />
<lib dir="../../../contrib/extraction/lib" regex=".*\.jar" />
<lib dir="../../../dist/" regex="apache-solr-clustering-\d.*\.jar" />
<lib dir="../../../contrib/clustering/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="apache-solr-langid-\d.*\.jar" />
<lib dir="../../../contrib/langid/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="apache-solr-velocity-\d.*\.jar" />
<lib dir="../../../contrib/velocity/lib" regex=".*\.jar" />
<!-- If a 'dir' option (with or without a regex) is used and nothing
is found that matches, it will be ignored
-->
<lib dir="/total/crap/dir/ignored" />
<!-- an exact 'path' can be used instead of a 'dir' to specify a
specific file. This will cause a serious error to be logged if
it can't be loaded.
-->
<!--
<lib path="../a-jar-that-does-not-exist.jar" />
-->
<!-- MDSolrDIHTransformer JARs inside -->
<!-- these are at C:\AbcData\V12\application server\server\default\lib -->
<lib path="../../../../default/lib/common.jar" />
<lib path="../../../../default/lib/webapp.jar" />
<!-- <lib path="../../../default/lib/mdAddr.jar" /> -->
<lib path="../../../../default/lib/commons-pool-1.4.jar" />
<!-- Data Directory
Used to specify an alternate directory to hold all index data
other than the default ./data under the Solr home. If
replication is in use, this should match the replication
configuration.
-->
<dataDir>${solr.data.dir:}</dataDir>
<!-- The DirectoryFactory to use for indexes.
solr.StandardDirectoryFactory is filesystem
based and tries to pick the best implementation for the current
JVM and platform. solr.NRTCachingDirectoryFactory, the default,
wraps solr.StandardDirectoryFactory and caches small files in memory
for better NRT performance.
One can force a particular implementation via
solr.MMapDirectoryFactory,
solr.NIOFSDirectoryFactory, or solr.SimpleFSDirectoryFactory.
solr.RAMDirectoryFactory is memory based, not
persistent, and doesn't work with replication.
-->
<!-- <directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/> -->
<directoryFactory name="DirectoryFactory"
class="org.apache.solr.core.StandardDirectoryFactory" />
<!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Index Config - These settings control low-level behavior of indexing
Most example settings here show the default value, but are commented
out, to more easily see where customizations have been made.
Note: This replaces <indexDefaults> and <mainIndex> from older
versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
<indexConfig>
<!-- maxFieldLength was removed in 4.0. To get similar behavior,
include a
LimitTokenCountFilterFactory in your fieldType definition. E.g.
<filter class="solr.LimitTokenCountFilterFactory"
maxTokenCount="10000"/>
-->
<!-- Maximum time to wait for a write lock (ms) for an IndexWriter.
Default: 1000 -->
<!-- <writeLockTimeout>1000</writeLockTimeout> -->
<!-- Expert: Enabling compound file will use less files for the index,
using fewer file descriptors on the expense of performance
decrease.
Default in Lucene is "true". Default in Solr is "false" (since
3.6) -->
<!-- <useCompoundFile>false</useCompoundFile> -->
<!-- ramBufferSizeMB sets the amount of RAM that may be used by Lucene
indexing for buffering added documents and deletions before they
are
flushed to the Directory.
maxBufferedDocs sets a limit on the number of documents buffered
before flushing.
If both ramBufferSizeMB and maxBufferedDocs is set, then
Lucene will flush based on whichever limit is hit first. -->
<!-- <ramBufferSizeMB>32</ramBufferSizeMB> -->
<!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
<!-- Expert: Merge Policy
The Merge Policy in Lucene controls how merging of segments is
done.
The default since Solr/Lucene 3.3 is TieredMergePolicy.
The default since Lucene 2.3 was the LogByteSizeMergePolicy,
Even older versions of Lucene used LogDocMergePolicy.
-->
<!--
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">10</int>
<int name="segmentsPerTier">10</int>
</mergePolicy>
-->
<!-- Merge Factor
The merge factor controls how many segments will get merged at a
time.
For TieredMergePolicy, mergeFactor is a convenience parameter which
will set both MaxMergeAtOnce and SegmentsPerTier at once.
For LogByteSizeMergePolicy, mergeFactor decides how many new
segments
will be allowed before they are merged into one.
Default is 10 for both merge policies.
-->
<!--
<mergeFactor>10</mergeFactor>
-->
<!-- Expert: Merge Scheduler
The Merge Scheduler in Lucene controls how merges are
performed. The ConcurrentMergeScheduler (Lucene 2.3 default)
can perform merges in the background using separate threads.
The SerialMergeScheduler (Lucene 2.2 default) does not.
-->
<!--
<mergeScheduler
class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
-->
<!-- LockFactory
This option specifies which Lucene LockFactory implementation
to use.
single = SingleInstanceLockFactory - suggested for a
read-only index or when there is no possibility of
another process trying to modify the index.
native = NativeFSLockFactory - uses OS native file locking.
Do not use when multiple solr webapps in the same
JVM are attempting to share a single index.
simple = SimpleFSLockFactory - uses a plain file for locking
Defaults: 'native' is default for Solr3.6 and later, otherwise
'simple' is the default
More details on the nuances of each LockFactory...
http://wiki.apache.org/lucene-java/AvailableLockFactories
-->
<!-- <lockType>native</lockType> -->
<!-- Unlock On Startup
If true, unlock any held write or commit locks on startup.
This defeats the locking mechanism that allows multiple
processes to safely access a lucene index, and should be used
with care. Default is "false".
This is not needed if lock type is 'none' or 'single'
-->
<!--
<unlockOnStartup>false</unlockOnStartup>
-->
<!-- Expert: Controls how often Lucene loads terms into memory
Default is 128 and is likely good for most everyone.
-->
<!-- <termIndexInterval>128</termIndexInterval> -->
<!-- If true, IndexReaders will be reopened (often more efficient)
instead of closed and then opened. Default: true
-->
<!--
<reopenReaders>true</reopenReaders>
-->
<!-- Commit Deletion Policy
Custom deletion policies can be specified here. The class must
implement org.apache.lucene.index.IndexDeletionPolicy.
http://lucene.apache.org/java/3_5_0/api/core/org/apache/lucene/index/IndexDeletionPolicy.html
The default Solr IndexDeletionPolicy implementation supports
deleting index commit points on number of commits, age of
commit point and optimized status.
The latest commit point should always be preserved regardless
of the criteria.
-->
<!--
<deletionPolicy class="solr.SolrDeletionPolicy">
-->
<!-- The number of commit points to be kept -->
<!-- <str name="maxCommitsToKeep">1</str> -->
<!-- The number of optimized commit points to be kept -->
<!-- <str name="maxOptimizedCommitsToKeep">0</str> -->
<!--
Delete all commit points once they have reached the given age.
Supports DateMathParser syntax e.g.
-->
<!--
<str name="maxCommitAge">30MINUTES</str>
<str name="maxCommitAge">1DAY</str>
-->
<!--
</deletionPolicy>
-->
<!-- Lucene Infostream
To aid in advanced debugging, Lucene provides an "InfoStream"
of detailed information when indexing.
Setting The value to true will instruct the underlying Lucene
IndexWriter to write its debugging info the specified file
-->
<!-- <infoStream file="INFOSTREAM.txt">false</infoStream> -->
</indexConfig>
<!-- JMX
This example enables JMX if and only if an existing MBeanServer
is found, use this if you want to configure JMX through JVM
parameters. Remove this to disable exposing Solr configuration
and statistics to JMX.
For more details see http://wiki.apache.org/solr/SolrJmx
-->
<!-- this is in Solr-4.0, but not 3.6.1 <jmx /> -->
<!-- If you want to connect to a particular server, specify the
agentId
-->
<!-- <jmx agentId="myAgent" /> -->
<!-- If you want to start a new MBeanServer, specify the serviceUrl -->
<!-- <jmx serviceUrl="service:jmx:rmi:///jndi/rmi://localhost:9999/solr"/>
-->
<!-- The default high-performance update handler -->
<updateHandler class="solr.DirectUpdateHandler2">
<!-- AutoCommit
Perform a hard commit automatically under certain conditions.
Instead of enabling autoCommit, consider using "commitWithin"
when adding documents.
http://wiki.apache.org/solr/UpdateXmlMessages
maxDocs - Maximum number of documents to add since the last
commit before automatically triggering a new commit.
maxTime - Maximum amount of time in ms that is allowed to pass
since a document was added before automaticly
triggering a new commit.
openSearcher - if false, the commit causes recent index changes
to be flushed to stable storage, but does not cause a new
searcher to be opened to make those changes visible.
-->
<!-- <autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
-->
<!-- softAutoCommit is like autoCommit except it causes a
'soft' commit which only ensures that changes are visible
but does not ensure that data is synced to disk. This is
faster and more near-realtime friendly than a hard commit.
-->
<!--
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
-->
<!-- Update Related Event Listeners
Various IndexWriter related events can trigger Listeners to
take actions.
postCommit - fired after every commit or optimize command
postOptimize - fired after every optimize command
-->
<!-- The RunExecutableListener executes an external command from a
hook such as postCommit or postOptimize.
exe - the name of the executable to run
dir - dir to use as the current working directory. (default=".")
wait - the calling thread waits until the executable returns.
(default="true")
args - the arguments to pass to the program. (default is none)
env - environment variables to set. (default is none)
-->
<!-- This example shows how RunExecutableListener could be used
with the script based replication...
http://wiki.apache.org/solr/CollectionDistribution
-->
<!--
<listener event="postCommit" class="solr.RunExecutableListener">
<str name="exe">solr/bin/snapshooter</str>
<str name="dir">.</str>
<bool name="wait">true</bool>
<arr name="args"> <str>arg1</str> <str>arg2</str> </arr>
<arr name="env"> <str>MYVAR=val1</str> </arr>
</listener>
-->
<!-- Enables a transaction log, currently used for real-time get.
"dir" - the target directory for transaction logs, defaults to the
solr data directory. -->
<!--
<updateLog>
<str name="dir">${solr.data.dir:}</str>
</updateLog>
-->
</updateHandler>
<!-- IndexReaderFactory
Use the following format to specify a custom IndexReaderFactory,
which allows for alternate IndexReader implementations.
** Experimental Feature **
Please note - Using a custom IndexReaderFactory may prevent
certain other features from working. The API to
IndexReaderFactory may change without warning or may even be
removed from future releases if the problems cannot be
resolved.
** Features that may not work with custom IndexReaderFactory **
The ReplicationHandler assumes a disk-resident index. Using a
custom IndexReader implementation may cause incompatibility
with ReplicationHandler and may cause replication to not work
correctly. See SOLR-1366 for details.
-->
<!--
<indexReaderFactory name="IndexReaderFactory" class="package.class">
<str name="someArg">Some Value</str>
</indexReaderFactory >
-->
<!-- By explicitly declaring the Factory, the termIndexDivisor can
be specified.
-->
<!--
<indexReaderFactory name="IndexReaderFactory"
class="solr.StandardIndexReaderFactory">
<int name="setTermIndexDivisor">12</int>
</indexReaderFactory >
-->
<!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Query section - these settings control query time things like caches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
<query>
<!-- Max Boolean Clauses
Maximum number of clauses in each BooleanQuery, an exception
is thrown if exceeded.
** WARNING **
This option actually modifies a global Lucene property that
will affect all SolrCores. If multiple solrconfig.xml files
disagree on this property, the value at any given moment will
be based on the last SolrCore to be initialized.
-->
<maxBooleanClauses>1024</maxBooleanClauses>
<!-- Solr Internal Query Caches
There are two implementations of cache available for Solr,
LRUCache, based on a synchronized LinkedHashMap, and
FastLRUCache, based on a ConcurrentHashMap.
FastLRUCache has faster gets and slower puts in single
threaded operation and thus is generally faster than LRUCache
when the hit ratio of the cache is high (> 75%), and may be
faster under other scenarios on multi-cpu systems.
-->
<!-- Filter Cache
Cache used by SolrIndexSearcher for filters (DocSets),
unordered sets of *all* documents that match a query. When a
new searcher is opened, its caches may be prepopulated or
"autowarmed" using data from caches in the old searcher.
autowarmCount is the number of items to prepopulate. For
LRUCache, the autowarmed items will be the most recently
accessed items.
Parameters:
class - the SolrCache implementation LRUCache or
(LRUCache or FastLRUCache)
size - the maximum number of entries in the cache
initialSize - the initial capacity (number of entries) of
the cache. (see java.util.HashMap)
autowarmCount - the number of entries to prepopulate from
and old cache.
-->
<filterCache class="solr.FastLRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<!-- Query Result Cache
Caches results of searches - ordered lists of document ids
(DocList) based on a query, a sort, and the range of documents
requested.
-->
<queryResultCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<!-- Document Cache
Caches Lucene Document objects (the stored fields for each
document). Since Lucene internal document ids are transient,
this cache will not be autowarmed.
-->
<documentCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<!-- Field Value Cache
Cache used to hold field values that are quickly accessible
by document id. The fieldValueCache is created by default
even if not configured here.
-->
<!--
<fieldValueCache class="solr.FastLRUCache"
size="512"
autowarmCount="128"
showItems="32" />
-->
<!-- Custom Cache
Example of a generic cache. These caches may be accessed by
name through SolrIndexSearcher.getCache(),cacheLookup(), and
cacheInsert(). The purpose is to enable easy caching of
user/application level data. The regenerator argument should
be specified as an implementation of solr.CacheRegenerator
if autowarming is desired.
-->
<!--
<cache name="myUserCache"
class="solr.LRUCache"
size="4096"
initialSize="1024"
autowarmCount="1024"
regenerator="com.mycompany.MyRegenerator"
/>
-->
<!-- Lazy Field Loading
If true, stored fields that are not requested will be loaded
lazily. This can result in a significant speed improvement
if the usual case is to not load all stored fields,
especially if the skipped fields are large compressed text
fields.
-->
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<!-- Use Filter For Sorted Query
A possible optimization that attempts to use a filter to
satisfy a search. If the requested sort does not include
score, then the filterCache will be checked for a filter
matching the query. If found, the filter will be used as the
source of document ids, and then the sort will be applied to
that.
For most situations, this will not be useful unless you
frequently get the same search repeatedly with different sort
options, and none of them ever use "score"
-->
<!--
<useFilterForSortedQuery>true</useFilterForSortedQuery>
-->
<!-- Result Window Size
An optimization for use with the queryResultCache. When a search
is requested, a superset of the requested number of document ids
are collected. For example, if a search for a particular query
requests matching documents 10 through 19, and queryWindowSize is
50,
then documents 0 through 49 will be collected and cached. Any
further
requests in that range can be satisfied via the cache.
-->
<queryResultWindowSize>20</queryResultWindowSize>
<!-- Maximum number of documents to cache for any entry in the
queryResultCache.
-->
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<!-- Query Related Event Listeners
Various IndexSearcher related events can trigger Listeners to
take actions.
newSearcher - fired whenever a new searcher is being prepared
and there is a current searcher handling requests (aka
registered). It can be used to prime certain caches to
prevent long request times for certain requests.
firstSearcher - fired whenever a new searcher is being
prepared but there is no current registered searcher to handle
requests or to gain autowarming data from.
-->
<!-- QuerySenderListener takes an array of NamedList and executes a
local query request for each NamedList in sequence.
-->
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!--
<lst><str name="q">solr</str><str name="sort">price
asc</str></lst>
<lst><str name="q">rocks</str><str name="sort">weight
asc</str></lst>
-->
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">static firstSearcher warming in solrconfig.xml</str>
</lst>
</arr>
</listener>
<!-- Use Cold Searcher
If a search request comes in and there is no current
registered searcher, then immediately register the still
warming searcher and use it. If "false" then all requests
will block until the first searcher is done warming.
-->
<useColdSearcher>false</useColdSearcher>
<!-- Max Warming Searchers
Maximum number of searchers that may be warming in the
background concurrently. An error is returned if this limit
is exceeded.
Recommend values of 1-2 for read-only slaves, higher for
masters w/o cache warming.
-->
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<!-- Request Dispatcher
This section contains instructions for how the SolrDispatchFilter
should behave when processing requests for this SolrCore.
handleSelect is a legacy option that affects the behavior of requests
such as /select?qt=XXX
handleSelect="true" will cause the SolrDispatchFilter to process
the request and dispatch the query to a handler specified by the
"qt" param, assuming "/select" isn't already registered.
handleSelect="false" will cause the SolrDispatchFilter to
ignore "/select" requests, resulting in a 404 unless a handler
is explicitly registered with the name "/select"
handleSelect="true" is not recommended for new users, but is the
default
for backwards compatibility
-->
<requestDispatcher handleSelect="false" >
<!-- Request Parsing
These settings indicate how Solr Requests may be parsed, and
what restrictions may be placed on the ContentStreams from
those requests
enableRemoteStreaming - enables use of the stream.file
and stream.url parameters for specifying remote streams.
multipartUploadLimitInKB - specifies the max size of
Multipart File Uploads that Solr will allow in a Request.
*** WARNING ***
The settings below authorize Solr to fetch remote files, You
should make sure your system has some authentication before
using enableRemoteStreaming="true"
-->
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000" />
<!-- HTTP Caching
Set HTTP caching related parameters (for proxy caches and clients).
The options below instruct Solr not to output any HTTP Caching
related headers
-->
<httpCaching never304="true" />
<!-- If you include a <cacheControl> directive, it will be used to
generate a Cache-Control header (as well as an Expires header
if the value contains "max-age=")
By default, no Cache-Control header is generated.
You can use the <cacheControl> option even if you have set
never304="true"
-->
<!--
<httpCaching never304="true" >
<cacheControl>max-age=30, public</cacheControl>
</httpCaching>
-->
<!-- To enable Solr to respond with automatically generated HTTP
Caching headers, and to response to Cache Validation requests
correctly, set the value of never304="false"
This will cause Solr to generate Last-Modified and ETag
headers based on the properties of the Index.
The following options can also be specified to affect the
values of these headers...
lastModFrom - the default value is "openTime" which means the
Last-Modified value (and validation against If-Modified-Since
requests) will all be relative to when the current Searcher
was opened. You can change it to lastModFrom="dirLastMod" if
you want the value to exactly correspond to when the physical
index was last modified.
etagSeed="..." is an option you can change to force the ETag
header (and validation against If-None-Match requests) to be
different even if the index has not changed (ie: when making
significant changes to your config file)
(lastModifiedFrom and etagSeed are both ignored if you use
the never304="true" option)
-->
<!--
<httpCaching lastModifiedFrom="openTime"
etagSeed="Solr">
<cacheControl>max-age=30, public</cacheControl>
</httpCaching>
-->
</requestDispatcher>
<!-- DIH / Database / JDBC -->
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">database.xml</str>
</lst>
</requestHandler>
<!-- Request Handlers
http://wiki.apache.org/solr/SolrRequestHandler
Incoming queries will be dispatched to a specific handler by name
based on the path specified in the request.
Legacy behavior: If the request path uses "/select" but no Request
Handler has that name, and if handleSelect="true" has been specified
in
the requestDispatcher, then the Request Handler is dispatched based
on
the qt parameter. Handlers without a leading '/' are accessed this
way
like so: http://host/app/[core/]select?qt=name If no qt is
given, then the requestHandler that declares default="true" will be
used or the one named "standard".
If a Request Handler is declared with startup="lazy", then it will
not be initialized until the first request that uses it.
-->
<!-- SearchHandler
http://wiki.apache.org/solr/SearchHandler
For processing Search Queries, the primary Request Handler
provided with Solr is "SearchHandler" It delegates to a sequent
of SearchComponents (see below) and supports distributed
queries across multiple shards
-->
<requestHandler name="/select" class="solr.SearchHandler">
<!-- default values for query parameters can be specified, these
will be overridden by parameters in the request
-->
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
</lst>
<!-- In addition to defaults, "appends" params can be specified
to identify values which should be appended to the list of
multi-val params from the query (or the existing "defaults").
-->
<!-- In this example, the param "fq=instock:true" would be appended to
any query time fq params the user may specify, as a mechanism for
partitioning the index, independent of any user selected filtering
that may also be desired (perhaps as a result of faceted
searching).
NOTE: there is *absolutely* nothing a client can do to prevent
these
"appends" values from being used, so don't use this mechanism
unless you are sure you always want it.
-->
<!--
<lst name="appends">
<str name="fq">inStock:true</str>
</lst>
-->
<!-- "invariants" are a way of letting the Solr maintainer lock down
the options available to Solr clients. Any params values
specified here are used regardless of what values may be specified
in either the query, the "defaults", or the "appends" params.
In this example, the facet.field and facet.query params would
be fixed, limiting the facets clients can use. Faceting is
not turned on by default - but if the client does specify
facet=true in the request, these are the only facets they
will be able to see counts for; regardless of what other
facet.field or facet.query params they may specify.
NOTE: there is *absolutely* nothing a client can do to prevent
these
"invariants" values from being used, so don't use this mechanism
unless you are sure you always want it.
-->
<!--
<lst name="invariants">
<str name="facet.field">cat</str>
<str name="facet.field">manu_exact</str>
<str name="facet.query">price:[* TO 500]</str>
<str name="facet.query">price:[500 TO *]</str>
</lst>
-->
<!-- If the default list of SearchComponents is not desired, that
list can either be overridden completely, or components can be
prepended or appended to the default list. (see below)
-->
<!--
<arr name="components">
<str>nameOfCustomComponent1</str>
<str>nameOfCustomComponent2</str>
</arr>
-->
</requestHandler>
<!-- A request handler that returns indented JSON by default -->
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="df">text</str>
</lst>
</requestHandler>
<!-- realtime get handler, guaranteed to return the latest stored fields
of
any document, without the need to commit or open a new searcher. The
current implementation relies on the updateLog feature being
enabled. -->
<!-- Solr 4.0 only!
<requestHandler name="/get" class="solr.RealTimeGetHandler">
<lst name="defaults">
<str name="omitHeader">true</str>
<str name="wt">json</str>
<str name="indent">true</str>
</lst>
</requestHandler>
-->
<!-- A Robust Example
This example SearchHandler declaration shows off usage of the
SearchHandler with many defaults declared
Note that multiple instances of the same Request Handler
(SearchHandler) can be registered multiple times with different
names (and different init parameters)
-->
<requestHandler name="/browse" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<!-- VelocityResponseWriter settings -->
<str name="wt">velocity</str>
<str name="v.template">browse</str>
<str name="v.layout">layout</str>
<str name="title">Solritas</str>
<!-- Query settings -->
<str name="defType">edismax</str>
<str name="qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
title^10.0 description^5.0 keywords^5.0 author^2.0
resourcename^1.0
</str>
<str name="df">text</str>
<str name="mm">100%</str>
<str name="q.alt">*:*</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
<str name="mlt.qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
title^10.0 description^5.0 keywords^5.0 author^2.0 resourcename^1.0
</str>
<str
name="mlt.fl">text,features,name,sku,id,manu,cat,title,description,keywords,author,resourcename</str>
<int name="mlt.count">3</int>
<!-- Faceting defaults -->
<str name="facet">on</str>
<str name="facet.field">cat</str>
<str name="facet.field">manu_exact</str>
<str name="facet.field">content_type</str>
<str name="facet.field">author_s</str>
<str name="facet.query">ipod</str>
<str name="facet.query">GB</str>
<str name="facet.mincount">1</str>
<str name="facet.pivot">cat,inStock</str>
<str name="facet.range.other">after</str>
<str name="facet.range">price</str>
<int name="f.price.facet.range.start">0</int>
<int name="f.price.facet.range.end">600</int>
<int name="f.price.facet.range.gap">50</int>
<str name="facet.range">popularity</str>
<int name="f.popularity.facet.range.start">0</int>
<int name="f.popularity.facet.range.end">10</int>
<int name="f.popularity.facet.range.gap">3</int>
<str name="facet.range">manufacturedate_dt</str>
<str
name="f.manufacturedate_dt.facet.range.start">NOW/YEAR-10YEARS</str>
<str name="f.manufacturedate_dt.facet.range.end">NOW</str>
<str name="f.manufacturedate_dt.facet.range.gap">+1YEAR</str>
<str name="f.manufacturedate_dt.facet.range.other">before</str>
<str name="f.manufacturedate_dt.facet.range.other">after</str>
<!-- Highlighting defaults -->
<str name="hl">on</str>
<str name="hl.fl">content features title name</str>
<str name="hl.encoder">html</str>
<str name="hl.simple.pre"><b></str>
<str name="hl.simple.post"></b></str>
<str name="f.title.hl.fragsize">0</str>
<str name="f.title.hl.alternateField">title</str>
<str name="f.name.hl.fragsize">0</str>
<str name="f.name.hl.alternateField">name</str>
<str name="f.content.hl.snippets">3</str>
<str name="f.content.hl.fragsize">200</str>
<str name="f.content.hl.alternateField">content</str>
<str name="f.content.hl.maxAlternateFieldLength">750</str>
<!-- Spell checking defaults -->
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">false</str>
<str name="spellcheck.count">5</str>
<str name="spellcheck.alternativeTermCount">2</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">5</str>
<str name="spellcheck.maxCollations">3</str>
</lst>
<!-- append spellchecking to our list of components -->
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<!-- <valueSourceParser name="fakeStr"
class="com.abc.fulltext.multiop.FakeStrValueSourceParser" /> -->
<!-- Update Request Handler.
http://wiki.apache.org/solr/UpdateXmlMessages
The canonical Request Handler for Modifying the Index through
commands specified using XML, JSON, CSV, or JAVABIN
Note: Since solr1.1 requestHandlers requires a valid content
type header if posted in the body. For example, curl now
requires: -H 'Content-type:text/xml; charset=utf-8'
To override the request content type and force a specific
Content-type, use the request parameter:
?update.contentType=text/csv
This handler will pick a response format to match the input
if the 'wt' parameter is not explicit
-->
<!-- <requestHandler name="/update" class="solr.UpdateRequestHandler"> -->
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
<!-- See below for information on defining
updateRequestProcessorChains that can be used by name
on each Update Request
-->
<!--
<lst name="defaults">
<str name="update.chain">dedupe</str>
</lst>
-->
</requestHandler>
<!-- Solr Cell Update Request Handler
http://wiki.apache.org/solr/ExtractingRequestHandler
-->
<requestHandler name="/update/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.a">links</str>
<str name="fmap.div">ignored_</str>
</lst>
</requestHandler>
<!-- Field Analysis Request Handler
RequestHandler that provides much the same functionality as
analysis.jsp. Provides the ability to specify multiple field
types and field names in the same request and outputs
index-time and query-time analysis for each of them.
Request parameters are:
analysis.fieldname - field name whose analyzers are to be used
analysis.fieldtype - field type whose analyzers are to be used
analysis.fieldvalue - text for index-time analysis
q (or analysis.q) - text for query time analysis
analysis.showmatch (true|false) - When set to true and when
query analysis is performed, the produced tokens of the
field value analysis will be marked as "matched" for every
token that is produces by the query analysis
-->
<requestHandler name="/analysis/field"
startup="lazy"
class="solr.FieldAnalysisRequestHandler" />
<!-- Document Analysis Handler
http://wiki.apache.org/solr/AnalysisRequestHandler
An analysis handler that provides a breakdown of the analysis
process of provided documents. This handler expects a (single)
content stream with the following format:
<docs>
<doc>
<field name="id">1</field>
<field name="name">The Name</field>
<field name="text">The Text Value</field>
</doc>
<doc>...</doc>
<doc>...</doc>
...
</docs>
Note: Each document must contain a field which serves as the
unique key. This key is used in the returned response to associate
an analysis breakdown to the analyzed document.
Like the FieldAnalysisRequestHandler, this handler also supports
query analysis by sending either an "analysis.query" or "q"
request parameter that holds the query text to be analyzed. It
also supports the "analysis.showmatch" parameter which when set to
true, all field tokens that match the query tokens will be marked
as a "match".
-->
<requestHandler name="/analysis/document"
class="solr.DocumentAnalysisRequestHandler"
startup="lazy" />
<!-- Admin Handlers
Admin Handlers - This will register all the standard admin
RequestHandlers.
-->
<requestHandler name="/admin/"
class="solr.admin.AdminHandlers" />
<!-- This single handler is equivalent to the following... -->
<!--
<requestHandler name="/admin/luke"
class="solr.admin.LukeRequestHandler" />
<requestHandler name="/admin/system"
class="solr.admin.SystemInfoHandler" />
<requestHandler name="/admin/plugins"
class="solr.admin.PluginInfoHandler" />
<requestHandler name="/admin/threads"
class="solr.admin.ThreadDumpHandler" />
<requestHandler name="/admin/properties"
class="solr.admin.PropertiesRequestHandler" />
<requestHandler name="/admin/file"
class="solr.admin.ShowFileRequestHandler" >
-->
<!-- If you wish to hide files under ${solr.home}/conf, explicitly
register the ShowFileRequestHandler using:
-->
<!--
<requestHandler name="/admin/file"
class="solr.admin.ShowFileRequestHandler" >
<lst name="invariants">
<str name="hidden">synonyms.txt</str>
<str name="hidden">anotherfile.txt</str>
</lst>
</requestHandler>
-->
<!-- ping/healthcheck -->
<requestHandler name="/admin/ping" class="solr.PingRequestHandler">
<lst name="invariants">
<str name="q">solrpingquery</str>
</lst>
<lst name="defaults">
<str name="echoParams">all</str>
</lst>
<!-- An optional feature of the PingRequestHandler is to configure the
handler with a "healthcheckFile" which can be used to
enable/disable
the PingRequestHandler.
relative paths are resolved against the data dir
-->
<!-- <str name="healthcheckFile">server-enabled.txt</str> -->
</requestHandler>
<!-- Echo the request contents back to the client -->
<requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="echoHandler">true</str>
</lst>
</requestHandler>
<!-- Solr Replication
The SolrReplicationHandler supports replicating indexes from a
"master" used for indexing and "slaves" used for queries.
http://wiki.apache.org/solr/SolrReplication
In the example below, remove the <lst name="master"> section if
this is just a slave and remove the <lst name="slave"> section
if this is just a master.
-->
<!--
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">schema.xml,stopwords.txt</str>
</lst>
<lst name="slave">
<str name="masterUrl">http://localhost:8983/solr</str>
<str name="pollInterval">00:00:60</str>
</lst>
</requestHandler>
-->
<!-- Solr Replication for SolrCloud Recovery
This is the config need for SolrCloud's recovery replication.
-->
<requestHandler name="/replication" class="solr.ReplicationHandler"
startup="lazy" />
<!-- Search Components
Search components are registered to SolrCore and used by
instances of SearchHandler (which can access them by name)
By default, the following components are available:
<searchComponent name="query" class="solr.QueryComponent" />
<searchComponent name="facet" class="solr.FacetComponent" />
<searchComponent name="mlt" class="solr.MoreLikeThisComponent"
/>
<searchComponent name="highlight" class="solr.HighlightComponent" />
<searchComponent name="stats" class="solr.StatsComponent" />
<searchComponent name="debug" class="solr.DebugComponent" />
Default configuration in a requestHandler would look like:
<arr name="components">
<str>query</str>
<str>facet</str>
<str>mlt</str>
<str>highlight</str>
<str>stats</str>
<str>debug</str>
</arr>
If you register a searchComponent to one of the standard names,
that will be used instead of the default.
To insert components before or after the 'standard' components, use:
<arr name="first-components">
<str>myFirstComponentName</str>
</arr>
<arr name="last-components">
<str>myLastComponentName</str>
</arr>
NOTE: The component registered with the name "debug" will
always be executed after the "last-components"
-->
<!-- Spell Check
The spell check component can return a list of alternative spelling
suggestions.
http://wiki.apache.org/solr/SpellCheckComponent
-->
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<!-- Multiple "Spell Checkers" can be declared and used by this
component
-->
<!-- a spellchecker built from a field of the main index -->
<!-- Solr 4.0 only!!!
<lst name="spellchecker">
<str name="distanceMeasure">internal</str>
<float name="accuracy">0.5</float>
<int name="maxEdits">2</int>
<int name="minPrefix">1</int>
<int name="maxInspections">5</int>
<int name="minQueryLength">4</int>
<float name="maxQueryFrequency">0.01</float>
<float name="thresholdTokenFrequency">.01</float>
</lst> -->
<!-- a spellchecker that can break or combine words. See "/spell"
handler below for usage -->
<!-- Solr 4.0 only!!! <lst name="spellchecker">
<str name="name">wordbreak</str>
<str name="classname">solr.WordBreakSolrSpellChecker</str>
<str name="field">name</str>
<str name="combineWords">true</str>
<str name="breakWords">true</str>
<int name="maxChanges">10</int>
</lst>
-->
<!-- a spellchecker that uses a different distance measure -->
<!--
<lst name="spellchecker">
<str name="name">jarowinkler</str>
<str name="field">spell</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="distanceMeasure">
org.apache.lucene.search.spell.JaroWinklerDistance
</str>
</lst>
-->
<!-- a spellchecker that use an alternate comparator
comparatorClass be one of:
1. score (default)
2. freq (Frequency first, then score)
3. A fully qualified class name
-->
<!--
<lst name="spellchecker">
<str name="name">freq</str>
<str name="field">lowerfilt</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="comparatorClass">freq</str>
-->
<!-- A spellchecker that reads the list of words from a file -->
<!--
<lst name="spellchecker">
<str name="classname">solr.FileBasedSpellChecker</str>
<str name="name">file</str>
<str name="sourceLocation">spellings.txt</str>
<str name="characterEncoding">UTF-8</str>
<str name="spellcheckIndexDir">spellcheckerFile</str>
</lst>
-->
</searchComponent>
<!-- A request handler for demonstrating the spellcheck component.
NOTE: This is purely as an example. The whole purpose of the
SpellCheckComponent is to hook it into the request handler that
handles your normal user queries so that a separate request is
not needed to get suggestions.
IN OTHER WORDS, THERE IS REALLY GOOD CHANCE THE SETUP BELOW IS
NOT WHAT YOU WANT FOR YOUR PRODUCTION SYSTEM!
See http://wiki.apache.org/solr/SpellCheckComponent for details
on the request parameters.
-->
<requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">text</str>
<!-- Solr will use suggestions from both the 'default' spellchecker
and from the 'wordbreak' spellchecker and combine them.
collations (re-written queries) can include a combination of
corrections from both spellcheckers -->
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck.dictionary">wordbreak</str>
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<!-- Term Vector Component
http://wiki.apache.org/solr/TermVectorComponent
-->
<searchComponent name="tvComponent" class="solr.TermVectorComponent"/>
<!-- A request handler for demonstrating the term vector component
This is purely as an example.
In reality you will likely want to add the component to your
already specified request handlers.
-->
<requestHandler name="/tvrh" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">text</str>
<bool name="tv">true</bool>
</lst>
<arr name="last-components">
<str>tvComponent</str>
</arr>
</requestHandler>
<!-- Clustering Component
http://wiki.apache.org/solr/ClusteringComponent
You'll need to set the solr.cluster.enabled system property
when running solr to run with clustering enabled:
java -Dsolr.clustering.enabled=true -jar start.jar
-->
<searchComponent name="clustering"
enable="${solr.clustering.enabled:false}"
class="solr.clustering.ClusteringComponent" >
<!-- Declare an engine -->
<lst name="engine">
<!-- The name, only one can be named "default" -->
<str name="name">default</str>
<!-- Class name of Carrot2 clustering algorithm.
Currently available algorithms are:
* org.carrot2.clustering.lingo.LingoClusteringAlgorithm
* org.carrot2.clustering.stc.STCClusteringAlgorithm
*
org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm
See http://project.carrot2.org/algorithms.html for the
algorithm's characteristics.
-->
<str
name="carrot.algorithm">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>
<!-- Overriding values for Carrot2 default algorithm attributes.
For a description of all available attributes, see:
http://download.carrot2.org/stable/manual/#chapter.components.
Use attribute key as name attribute of str elements
below. These can be further overridden for individual
requests by specifying attribute key as request parameter
name and attribute value as parameter value.
-->
<str name="LingoClusteringAlgorithm.desiredClusterCountBase">20</str>
<!-- Location of Carrot2 lexical resources.
A directory from which to load Carrot2-specific stop words
and stop labels. Absolute or relative to Solr config directory.
If a specific resource (e.g. stopwords.en) is present in the
specified dir, it will completely override the corresponding
default one that ships with Carrot2.
For an overview of Carrot2 lexical resources, see:
http://download.carrot2.org/head/manual/#chapter.lexical-resources
-->
<str name="carrot.lexicalResourcesDir">clustering/carrot2</str>
<!-- The language to assume for the documents.
For a list of allowed values, see:
http://download.carrot2.org/stable/manual/#section.attribute.lingo.MultilingualClustering.defaultLanguage
-->
<str name="MultilingualClustering.defaultLanguage">ENGLISH</str>
</lst>
<lst name="engine">
<str name="name">stc</str>
<str
name="carrot.algorithm">org.carrot2.clustering.stc.STCClusteringAlgorithm</str>
</lst>
</searchComponent>
<!-- A request handler for demonstrating the clustering component
This is purely as an example.
In reality you will likely want to add the component to your
already specified request handlers.
-->
<requestHandler name="/clustering"
startup="lazy"
enable="${solr.clustering.enabled:false}"
class="solr.SearchHandler">
<lst name="defaults">
<bool name="clustering">true</bool>
<str name="clustering.engine">default</str>
<bool name="clustering.results">true</bool>
<!-- The title field -->
<str name="carrot.title">name</str>
<str name="carrot.url">id</str>
<!-- The field to cluster on -->
<str name="carrot.snippet">features</str>
<!-- produce summaries -->
<bool name="carrot.produceSummary">true</bool>
<!-- the maximum number of labels per cluster -->
<!--<int name="carrot.numDescriptions">5</int>-->
<!-- produce sub clusters -->
<bool name="carrot.outputSubClusters">false</bool>
<str name="defType">edismax</str>
<str name="qf">
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
</str>
<str name="q.alt">*:*</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
</lst>
<arr name="last-components">
<str>clustering</str>
</arr>
</requestHandler>
<!-- Terms Component
http://wiki.apache.org/solr/TermsComponent
A component to return terms and document frequency of those
terms
-->
<searchComponent name="terms" class="solr.TermsComponent"/>
<!-- A request handler for demonstrating the terms component -->
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
<!-- Query Elevation Component
http://wiki.apache.org/solr/QueryElevationComponent
a search component that enables you to configure the top
results for a given query regardless of the normal lucene
scoring.
-->
<searchComponent name="elevator" class="solr.QueryElevationComponent" >
<!-- pick a fieldType to analyze queries -->
<str name="queryFieldType">string</str>
<str name="config-file">elevate.xml</str>
</searchComponent>
<!-- A request handler for demonstrating the elevator component -->
<requestHandler name="/elevate" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="df">text</str>
</lst>
<arr name="last-components">
<str>elevator</str>
</arr>
</requestHandler>
<!-- Highlighting Component
http://wiki.apache.org/solr/HighlightingParameters
-->
<searchComponent class="solr.HighlightComponent" name="highlight">
<highlighting>
<!-- Configure the standard fragmenter -->
<!-- This could most likely be commented out in the "default" case -->
<fragmenter name="gap"
default="true"
class="solr.highlight.GapFragmenter">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<!-- A regular-expression-based fragmenter
(for sentence extraction)
-->
<fragmenter name="regex"
class="solr.highlight.RegexFragmenter">
<lst name="defaults">
<!-- slightly smaller fragsizes work better because of slop -->
<int name="hl.fragsize">70</int>
<!-- allow 50% slop on fragment sizes -->
<float name="hl.regex.slop">0.5</float>
<!-- a basic sentence pattern -->
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<!-- Configure the standard formatter -->
<formatter name="html"
default="true"
class="solr.highlight.HtmlFormatter">
<lst name="defaults">
<str name="hl.simple.pre"><![CDATA[<em>]]></str>
<str name="hl.simple.post"><![CDATA[</em>]]></str>
</lst>
</formatter>
<!-- Configure the standard encoder -->
<encoder name="html"
class="solr.highlight.HtmlEncoder" />
<!-- Configure the standard fragListBuilder -->
<fragListBuilder name="simple"
class="solr.highlight.SimpleFragListBuilder"/>
<!-- Configure the single fragListBuilder -->
<fragListBuilder name="single"
class="solr.highlight.SingleFragListBuilder"/>
<!-- Configure the weighted fragListBuilder -->
<!-- Solr 4.0 only!!!<fragListBuilder name="weighted"
default="true"
class="solr.highlight.WeightedFragListBuilder"/>
-->
<!-- default tag FragmentsBuilder -->
<fragmentsBuilder name="default"
default="true"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<!--
<lst name="defaults">
<str name="hl.multiValuedSeparatorChar">/</str>
</lst>
-->
</fragmentsBuilder>
<!-- multi-colored tag FragmentsBuilder -->
<fragmentsBuilder name="colored"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<lst name="defaults">
<str name="hl.tag.pre"><![CDATA[
<b style="background:yellow">,<b
style="background:lawgreen">,
<b style="background:aquamarine">,<b
style="background:magenta">,
<b style="background:palegreen">,<b
style="background:coral">,
<b style="background:wheat">,<b style="background:khaki">,
<b style="background:lime">,<b
style="background:deepskyblue">]]></str>
<str name="hl.tag.post"><![CDATA[</b>]]></str>
</lst>
</fragmentsBuilder>
<boundaryScanner name="default"
default="true"
class="solr.highlight.SimpleBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.maxScan">10</str>
<str name="hl.bs.chars">.,!? 	 </str>
</lst>
</boundaryScanner>
<boundaryScanner name="breakIterator"
class="solr.highlight.BreakIteratorBoundaryScanner">
<lst name="defaults">
<!-- type should be one of CHARACTER, WORD(default), LINE and
SENTENCE -->
<str name="hl.bs.type">WORD</str>
<!-- language and country are used when constructing Locale
object. -->
<!-- And the Locale object will be used when getting instance of
BreakIterator -->
<str name="hl.bs.language">en</str>
<str name="hl.bs.country">US</str>
</lst>
</boundaryScanner>
</highlighting>
</searchComponent>
<!-- Update Processors
Chains of Update Processor Factories for dealing with Update
Requests can be declared, and then used by name in Update
Request Processors
http://wiki.apache.org/solr/UpdateRequestProcessor
-->
<!-- Deduplication
An example dedup update processor that creates the "id" field
on the fly based on the hash code of some other fields. This
example has overwriteDupes set to false since we are using the
id field as the signatureField and Solr will maintain
uniqueness based on that anyway.
-->
<!--
<updateRequestProcessorChain name="dedupe">
<processor class="solr.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<str name="signatureField">id</str>
<bool name="overwriteDupes">false</bool>
<str name="fields">name,features,cat</str>
<str name="signatureClass">solr.processor.Lookup3Signature</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
-->
<!-- Language identification
This example update chain identifies the language of the incoming
documents using the langid contrib. The detected language is
written to field language_s. No field name mapping is done.
The fields used for detection are text, title, subject and
description,
making this example suitable for detecting languages form full-text
rich documents injected via ExtractingRequestHandler.
See more about langId at
http://wiki.apache.org/solr/LanguageDetection
-->
<!--
<updateRequestProcessorChain name="langid">
<processor
class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
<str name="langid.fl">text,title,subject,description</str>
<str name="langid.langField">language_s</str>
<str name="langid.fallback">en</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
-->
<!-- Script update processor
This example hooks in an update processor implemented using JavaScript.
See more about the script update processor at
http://wiki.apache.org/solr/ScriptUpdateProcessor
-->
<!--
<updateRequestProcessorChain name="script">
<processor class="solr.StatelessScriptUpdateProcessorFactory">
<str name="script">update-script.js</str>
<lst name="params">
<str name="config_param">example config parameter</str>
</lst>
</processor>
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
-->
<!-- Response Writers
http://wiki.apache.org/solr/QueryResponseWriter
Request responses will be written using the writer specified by
the 'wt' request parameter matching the name of a registered
writer.
The "default" writer is the default and will be used if 'wt' is
not specified in the request.
-->
<!-- The following response writers are implicitly configured unless
overridden...
-->
<!--
<queryResponseWriter name="xml"
default="true"
class="solr.XMLResponseWriter" />
<queryResponseWriter name="json" class="solr.JSONResponseWriter"/>
<queryResponseWriter name="python" class="solr.PythonResponseWriter"/>
<queryResponseWriter name="ruby" class="solr.RubyResponseWriter"/>
<queryResponseWriter name="php" class="solr.PHPResponseWriter"/>
<queryResponseWriter name="phps"
class="solr.PHPSerializedResponseWriter"/>
<queryResponseWriter name="csv" class="solr.CSVResponseWriter"/>
-->
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<!-- For the purposes of the tutorial, JSON responses are written as
plain text so that they are easy to read in *any* browser.
If you expect a MIME type of "application/json" just remove this
override.
-->
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
<!--
Custom response writers can be declared as needed...
-->
<queryResponseWriter name="velocity"
class="solr.VelocityResponseWriter" startup="lazy"/>
<!-- XSLT response writer transforms the XML output by any xslt file found
in Solr's conf/xslt directory. Changes to xslt files are checked for
every xsltCacheLifetimeSeconds.
-->
<queryResponseWriter name="xslt" class="solr.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>
<!-- Query Parsers
http://wiki.apache.org/solr/SolrQuerySyntax
Multiple QParserPlugins can be registered by name, and then
used in either the "defType" param for the QueryComponent (used
by SearchHandler) or in LocalParams
-->
<!-- example of registering a query parser -->
<!--
<queryParser name="myparser" class="com.mycompany.MyQParserPlugin"/>
-->
<!-- Function Parsers
http://wiki.apache.org/solr/FunctionQuery
Multiple ValueSourceParsers can be registered by name, and then
used as function names when using the "func" QParser.
-->
<!-- example of registering a custom function parser -->
<!--
<valueSourceParser name="myfunc"
class="com.mycompany.MyValueSourceParser" />
-->
<!-- Document Transformers
http://wiki.apache.org/solr/DocTransformers
-->
<!--
Could be something like:
<transformer name="db"
class="com.mycompany.LoadFromDatabaseTransformer" >
<int name="connection">jdbc://....</int>
</transformer>
To add a constant value to all docs, use:
<transformer name="mytrans2"
class="org.apache.solr.response.transform.ValueAugmenterFactory" >
<int name="value">5</int>
</transformer>
If you want the user to still be able to change it with
_value:something_ use this:
<transformer name="mytrans3"
class="org.apache.solr.response.transform.ValueAugmenterFactory" >
<double name="defaultValue">5</double>
</transformer>
If you are using the QueryElevationComponent, you may wish to mark
documents that get boosted. The
EditorialMarkerFactory will do exactly that:
<transformer name="qecBooster"
class="org.apache.solr.response.transform.EditorialMarkerFactory" />
-->
<!-- Legacy config for the admin interface -->
<admin>
<defaultQuery>*:*</defaultQuery>
</admin>
</config>
On Fri, Apr 4, 2014 at 5:00 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> schema.xml below
>
>
> <?xml version="1.0" encoding="UTF-8" ?>
>
> <!-- Abc / Def Note:
> Look for Abc tag to see changes we've made.
> Main sections/tags you'll want:
> <fields>
> <dynamicField ... >
> <uniqueKey>id</uniqueKey>
> <copyField ... >
> <types>
> -->
>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
>
> <!--
> This is the Solr schema file. This file should be named "schema.xml" and
> should be in the conf directory under the solr home
> (i.e. ./solr/conf/schema.xml by default)
> or located where the classloader for the Solr webapp can find it.
>
> This example schema is the recommended starting point for users.
> It should be kept correct and concise, usable out-of-the-box.
>
> For more information, on how to customize this file, please see
> http://wiki.apache.org/solr/SchemaXml
>
> PERFORMANCE NOTE: this schema includes many optional features and should
> not
> be used for benchmarking. To improve performance one could
> - set stored="false" for all fields possible (esp large fields) when you
> only need to search on the field but don't need to return the original
> value.
> - set indexed="false" if you don't need to search on the field, but only
> return the field as a result of searching on other indexed fields.
> - remove all unneeded copyField statements
> - for best index size and searching performance, set "index" to false
> for all general text fields, use copyField to copy them to the
> catchall "text" field, and use that for searching.
> - For maximum indexing performance, use the StreamingUpdateSolrServer
> java client.
> - Remember to run the JVM in server mode, and use a higher logging level
> that avoids logging every request
> -->
>
> <schema name="example" version="1.5">
> <!-- attribute "name" is the name of this schema and is only used for
> display purposes.
> version="x.y" is Solr's version number for the schema syntax and
> semantics. It should
> not normally be changed by applications.
> 1.0: multiValued attribute did not exist, all fields are
> multiValued by nature
> 1.1: multiValued attribute introduced, false by default
> 1.2: omitTermFreqAndPositions attribute introduced, true by default
> except for text fields.
> 1.3: removed optional field compress feature
> 1.4: default auto-phrase (QueryParser feature) to off
> 1.5: omitNorms defaults to true for primitive field types (int,
> float, boolean, string...)
> -->
>
> <fields>
> <!-- Valid attributes for fields:
> name: mandatory - the name for the field
> type: mandatory - the name of a field type from the
> <types> fieldType section
> indexed: true if this field should be indexed (searchable or sortable)
> stored: true if this field should be retrievable
> multiValued: true if this field may contain multiple values per
> document
> omitNorms: (expert) set to true to omit the norms associated with
> this field (this disables length normalization and index-time
> boosting for the field, and saves some memory). Only full-text
> fields or fields that need an index-time boost need norms.
> Norms are omitted for primitive (non-analyzed) types by default.
> termVectors: [false] set to true to store the term vector for a
> given field.
> When using MoreLikeThis, fields used for similarity should be
> stored for best performance.
> termPositions: Store position information with the term vector.
> This will increase storage costs.
> termOffsets: Store offset information with the term vector. This
> will increase storage costs.
> required: The field is required. It will throw an error if the
> value does not exist
> default: a value that should be used if no value is specified
> when adding a document.
> -->
>
> <!-- field names should consist of alphanumeric or underscore
> characters only and
> not start with a digit. This is not currently strictly enforced,
> but other field names will not have first class support from all
> components
> and back compatibility is not guaranteed. Names with both leading
> and
> trailing underscores (e.g. _version_) are reserved.
> -->
>
> <field name="id" type="string" indexed="true" stored="true"
> required="true" multiValued="false" />
> <field name="sku" type="text_en_splitting_tight" indexed="true"
> stored="true" omitNorms="true"/>
> <field name="name" type="text_general" indexed="true" stored="true"/>
> <field name="manu" type="text_general" indexed="true" stored="true"
> omitNorms="true"/>
> <field name="cat" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="features" type="text_general" indexed="true" stored="true"
> multiValued="true"/>
> <field name="includes" type="text_general" indexed="true" stored="true"
> termVectors="true" termPositions="true" termOffsets="true" />
>
> <field name="weight" type="float" indexed="true" stored="true"/>
> <field name="price" type="float" indexed="true" stored="true"/>
> <field name="popularity" type="int" indexed="true" stored="true" />
> <field name="inStock" type="boolean" indexed="true" stored="true" />
>
> <field name="store" type="location" indexed="true" stored="true"/>
>
> <!-- Common metadata fields, named specifically to match up with
> SolrCell metadata when parsing rich documents such as Word, PDF.
> Some fields are multiValued only because Tika currently may return
> multiple values for them. Some metadata is parsed from the documents,
> but there are some which come from the client context:
> "content_type": From the HTTP headers of incoming stream
> "resourcename": From SolrCell request param resource.name
> -->
> <field name="title" type="text_general" indexed="true" stored="true"
> multiValued="true"/>
> <field name="subject" type="text_general" indexed="true" stored="true"/>
> <field name="description" type="text_general" indexed="true"
> stored="true"/>
> <field name="comments" type="text_general" indexed="true"
> stored="true"/>
> <field name="author" type="text_general" indexed="true" stored="true"/>
> <field name="keywords" type="text_general" indexed="true"
> stored="true"/>
> <field name="category" type="text_general" indexed="true"
> stored="true"/>
> <field name="resourcename" type="text_general" indexed="true"
> stored="true"/>
> <field name="url" type="text_general" indexed="true" stored="true"/>
> <field name="content_type" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="last_modified" type="date" indexed="true" stored="true"/>
> <field name="links" type="string" indexed="true" stored="true"
> multiValued="true"/>
>
> <!-- Main body of document extracted by SolrCell.
> NOTE: This field is not indexed by default, since it is also
> copied to "text"
> using copyField below. This is to save space. Use this field for
> returning and
> highlighting document content. Use the "text" field to search the
> content. -->
> <field name="content" type="text_general" indexed="false" stored="true"
> multiValued="true"/>
>
>
> <!-- catchall field, containing all other searchable text fields
> (implemented
> via copyField further on in this schema -->
> <field name="text" type="text_general" indexed="true" stored="false"
> multiValued="true"/>
>
> <!-- catchall text field that indexes tokens both normally and in
> reverse for efficient
> leading wildcard queries. -->
> <field name="text_rev" type="text_general_rev" indexed="true"
> stored="false" multiValued="true"/>
>
> <!-- non-tokenized version of manufacturer to make it easier to sort or
> group
> results by manufacturer. copied from "manu" via copyField -->
> <field name="manu_exact" type="string" indexed="true" stored="false"/>
>
> <field name="payloads" type="payloads" indexed="true" stored="true"/>
>
> <field name="_version_" type="long" indexed="true" stored="true"/>
>
>
> <!-- Uncommenting the following will create a "timestamp" field using
> a default value of "NOW" to indicate when each document was
> indexed.
> -->
> <field name="timestamp" type="date" indexed="true" stored="true"
> default="NOW" multiValued="false"/>
>
> <!-- Dynamic field definitions allow using convention over configuration
> for fields via the specification of patterns to match field names.
> EXAMPLE: name="*_i" will match any field ending in _i (like
> myid_i, z_i)
> RESTRICTION: the glob-like pattern in the name attribute must have
> a "*" only at the start or the end. -->
>
> <!-- Abc Automatic Fields -->
> <!-- ===================== -->
>
> <!-- Variants Stored -->
>
> <!-- stopwords renamed to percentmatchwholefieldcorewordsonly -->
> <dynamicField
>
> name="*percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> type="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
> indexed="true" stored="true"
> multiValued="true"
> />
> <!--<dynamicField
> name="*stopwords.porterstem.ignorenonalphanumeric.soundex_abc"
> type="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
> indexed="true" stored="true"/>-->
>
> <dynamicField
>
> name="*percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> type="abc_stopwords_porterstem_ignorenonalphanumeric"
> indexed="true" stored="false"
> />
> <!--<dynamicField name="*stopwords.porterstem.ignorenonalphanumeric_abc"
> type="abc_stopwords_porterstem_ignorenonalphanumeric" indexed="true"
> stored="false"/>-->
>
>
> <dynamicField name="*porterstem.ignorenonalphanumeric.soundex_abc"
> type="abc_porterstem_ignorenonalphanumeric_soundex" indexed="true"
> stored="false" multiValued="true" />
> <dynamicField name="*porterstem.ignorenonalphanumeric_abc"
> type="abc_porterstem_ignorenonalphanumeric" indexed="true" stored="false"/>
>
>
>
> <dynamicField name="*ignorecase_abc" type="abc_ignorecase"
> indexed="true" stored="false"/>
>
> <dynamicField name="*ignorepunctuation.soundex_abc"
> type="abc_ignorepunctuation_soundex" indexed="true" stored="false"/>
> <dynamicField name="*ignorepunctuation_abc"
> type="abc_ignorepunctuation" indexed="true" stored="false"/>
>
> <dynamicField name="*soundex_abc" type="abc_soundex" indexed="true"
> stored="false" multiValued="true" />
> <dynamicField name="*firstfivedigits_abc" type="abc_firstfivedigits"
> indexed="true" stored="false"/>
>
> <dynamicField name="*ignorenonalphanumeric.soundex_abc"
> type="abc_ignorenonalphanumeric_soundex" indexed="true" stored="false"/>
> <dynamicField name="*ignorenonalphanumeric_abc"
> type="abc_ignorenonalphanumeric" indexed="true" stored="false"/>
>
> <dynamicField name="*typo_abc" type="abc_typo" indexed="true"
> stored="false"/>
>
>
> <dynamicField name="*porterstem.soundex_abc"
> type="abc_porterstem_soundex" indexed="true" stored="false"/>
> <dynamicField name="*porterstem_abc" type="abc_porterstem"
> indexed="true" stored="false"/>
>
> <!-- Abc stopwords dynamicField -->
> <dynamicField name="*stopwords_abc" type="abc_stopwords" indexed="true"
> stored="false"/>
> <dynamicField name="*percentmatchwholefieldcorewordsonly.soundex_abc"
> type="abc_stopwords_soundex" indexed="true" stored="false"/>
> <dynamicField name="*percentmatchwholefieldcorewordsonly_abc"
> type="abc_stopwords" indexed="true" stored="false"/>
>
>
> <dynamicField name="*junk_abc" type="abc_junk" indexed="true"
> stored="false"/>
> <dynamicField name="*profanity_abc" type="abc_profanity" indexed="true"
> stored="false"/>
> <!-- Abc Thesaurus dynamicFields, stored -->
> <dynamicField name="*areacodethes_abc" type="abc_areacodethes"
> indexed="true" stored="false"/>
>
> <dynamicField name="*companynamesthes.soundex_abc"
> type="abc_companynamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*companynamesthes_abc" type="abc_companynamesthes"
> indexed="true" stored="false"/>
>
> <dynamicField name="*firstnamesthes.soundex_abc"
> type="abc_firstnamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*firstnamesthes_abc" type="abc_firstnamesthes"
> indexed="true" stored="false"/>
>
> <!-- Technically these are "address related" -->
> <dynamicField name="*citynamesthes.soundex_abc"
> type="abc_citynamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*citynamesthes_abc" type="abc_citynamesthes"
> indexed="true" stored="false"/>
>
> <dynamicField name="*statenamesthes.soundex_abc"
> type="abc_statenamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*statenamesthes_abc" type="abc_statenamesthes"
> indexed="true" stored="false"/>
>
> <dynamicField name="*provincenamesthes.soundex_abc"
> type="abc_provincenamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*provincenamesthes_abc"
> type="abc_provincenamesthes" indexed="true" stored="false"/>
>
> <dynamicField name="*countrynamesthes.soundex_abc"
> type="abc_countrynamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*countrynamesthes_abc" type="abc_countrynamesthes"
> indexed="true" stored="false"/>
>
> <!-- Also streetnamethes defined with Address fields -->
>
>
> <!--
> *********************************************
> Addresses: dynamicFields
> *********************************************
> streetname
> streetnumber
> streetsecondary
> streetpredirectional
> streetpostdirectional
> streetsuffix
> pobox
> TODO: ? matchnullone matchnullboth
> -->
> <!-- streetname / Street Name -->
> <dynamicField name="*streetnamepercentage.soundex_abc"
> type="abc_streetnamepercentage_soundex" indexed="true" stored="false"/>
> <dynamicField name="*streetnamepercentage_abc"
> type="abc_streetnamepercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetnamepercentage_nvl_abc"
> type="abc_nvl_streetnamepercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetnamepercentage.typo_abc"
> type="abc_streetnamepercentage.typo" indexed="true" stored="false"/>
> <dynamicField name="*streetnamepercentage.typo_nvl_abc"
> type="abc_nvl_streetnamepercentage.typo" indexed="true" stored="false"/>
> <dynamicField name="*streetnamematchnullone_abc"
> type="abc_streetnamematchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetnamematchnullone_nvl_abc"
> type="abc_nvl_streetnamematchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetnamematchnullboth_abc"
> type="abc_streetnamematchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*streetnamematchnullboth_nvl_abc"
> type="abc_nvl_streetnamematchnullboth" indexed="true" stored="false"/>
>
> <dynamicField name="*streetnamesthes.soundex_abc"
> type="abc_streetnamesthes_soundex" indexed="true" stored="false"/>
> <dynamicField name="*streetnamesthes_abc" type="abc_streetnamesthes"
> indexed="true" stored="false"/>
> <dynamicField name="*streetnamesthes_nvl_abc"
> type="abc_nvl_streetnamesthes" indexed="true" stored="false"/>
>
> <!-- streetnumber / Street Number -->
> <dynamicField name="*streetnumberpercentage_abc"
> type="abc_streetnumberpercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetnumberpercentage_nvl_abc"
> type="abc_nvl_streetnumberpercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetnumbermatchnullone_abc"
> type="abc_streetnumbermatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetnumbermatchnullone_nvl_abc"
> type="abc_nvl_streetnumbermatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetnumbermatchnullboth_abc"
> type="abc_streetnumbermatchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*streetnumbermatchnullboth_nvl_abc"
> type="abc_nvl_streetnumbermatchnullboth" indexed="true" stored="false"/>
>
> <!-- streetsecondary / Street Secondary -->
> <dynamicField name="*streetsecondarypercentage_abc"
> type="abc_streetsecondarypercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetsecondarypercentage_nvl_abc"
> type="abc_nvl_streetsecondarypercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetsecondarymatchnullone_abc"
> type="abc_streetsecondarymatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetsecondarymatchnullone_nvl_abc"
> type="abc_nvl_streetsecondarymatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetsecondarymatchnullboth_abc"
> type="abc_streetsecondarymatchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*streetsecondarymatchnullboth_nvl_abc"
> type="abc_nvl_streetsecondarymatchnullboth" indexed="true" stored="false"/>
>
> <!-- streetpredirectional / Street Predirectional / Street Pre Directional
> -->
> <dynamicField name="*streetpredirectionalpercentage_abc"
> type="abc_streetpredirectionalpercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetpredirectionalpercentage_nvl_abc"
> type="abc_nvl_streetpredirectionalpercentage" indexed="true"
> stored="false"/>
> <dynamicField name="*streetpredirectionalmatchnullone_abc"
> type="abc_streetpredirectionalmatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetpredirectionalmatchnullone_nvl_abc"
> type="abc_nvl_streetpredirectionalmatchnullone" indexed="true"
> stored="false"/>
> <dynamicField name="*streetpredirectionalmatchnullboth_abc"
> type="abc_streetpredirectionalmatchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*streetpredirectionalmatchnullboth_nvl_abc"
> type="abc_nvl_streetpredirectionalmatchnullboth" indexed="true"
> stored="false"/>
>
> <!-- streetpostdirectional / Street Postdirectional / Street Post
> Directional -->
> <dynamicField name="*streetpostdirectionalpercentage_abc"
> type="abc_streetpostdirectionalpercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetpostdirectionalpercentage_nvl_abc"
> type="abc_nvl_streetpostdirectionalpercentage" indexed="true"
> stored="false"/>
> <dynamicField name="*streetpostdirectionalmatchnullone_abc"
> type="abc_streetpostdirectionalmatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetpostdirectionalmatchnullone_nvl_abc"
> type="abc_nvl_streetpostdirectionalmatchnullone" indexed="true"
> stored="false"/>
> <dynamicField name="*streetpostdirectionalmatchnullboth_abc"
> type="abc_streetpostdirectionalmatchnullboth" indexed="true"
> stored="false"/>
> <dynamicField name="*streetpostdirectionalmatchnullboth_nvl_abc"
> type="abc_nvl_streetpostdirectionalmatchnullboth" indexed="true"
> stored="false"/>
>
> <!-- streetsuffix / Street Suffix -->
> <dynamicField name="*streetsuffixpercentage_abc"
> type="abc_streetsuffixpercentage" indexed="true" stored="false"/>
> <dynamicField name="*streetsuffixpercentage_nvl_abc"
> type="abc_nvl_streetsuffixpercentage" indexed="true" stored="false"/>
>
> <dynamicField name="*streetsuffixmatchnullone_abc"
> type="abc_streetsuffixmatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*streetsuffixmatchnullone_nvl_abc"
> type="abc_nvl_streetsuffixmatchnullone" indexed="true" stored="false"/>
>
> <dynamicField name="*streetsuffixmatchnullboth_abc"
> type="abc_streetsuffixmatchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*streetsuffixmatchnullboth_nvl_abc"
> type="abc_nvl_streetsuffixmatchnullboth" indexed="true" stored="false"/>
>
> <!-- pobox / PO Box -->
> <dynamicField name="*poboxpercentage_abc" type="abc_poboxpercentage"
> indexed="true" stored="false"/>
> <dynamicField name="*poboxpercentage_nvl_abc"
> type="abc_nvl_poboxpercentage" indexed="true" stored="false"/>
> <dynamicField name="*poboxmatchnullone_abc" type="abc_poboxmatchnullone"
> indexed="true" stored="false"/>
> <dynamicField name="*poboxmatchnullone_nvl_abc"
> type="abc_nvl_poboxmatchnullone" indexed="true" stored="false"/>
> <dynamicField name="*poboxmatchnullboth_abc"
> type="abc_poboxmatchnullboth" indexed="true" stored="false"/>
> <dynamicField name="*poboxmatchnullboth_nvl_abc"
> type="abc_nvl_poboxmatchnullboth" indexed="true" stored="false"/>
>
> <!-- matchnullone matchnullboth / Match Null One / Match Null Both -->
> <!-- TODO: not sure we still need these? Also in fieldType section. -->
> <dynamicField name="*matchnullone_abc" type="abc_matchnullone"
> indexed="true" stored="false"/>
> <dynamicField name="*matchnullone_nvl_abc" type="abc_nvl_matchnullone"
> indexed="true" stored="false"/>
> <dynamicField name="*matchnullboth_abc" type="abc_matchnullboth"
> indexed="true" stored="false"/>
> <dynamicField name="*matchnullboth_nvl_abc" type="abc_nvl_matchnullboth"
> indexed="true" stored="false"/>
>
>
> <!-- Fields we're sure are text -->
> <dynamicField name="*fulltext_abc" type="abc_fulltext" indexed="true"
> stored="true"/>
> <dynamicField name="*id_abc" type="abc_id" indexed="true"
> stored="true"/>
>
> <!-- This was the early v4 Solr behavior, but now whole-field is
> default -->
>
> <!-- with stop words -->
> <dynamicField name="*percentmatchanywordcorewordsonly.soundex_abc"
> type="abc_word_based_stopwords_soundex" indexed="true" stored="true"/>
> <dynamicField name="*percentmatchanywordcorewordsonly_nvl_abc"
> type="abc_word_based_stopwords_nvl" indexed="true" stored="true"/>
> <dynamicField name="*percentmatchanywordcorewordsonly_abc"
> type="abc_word_based_stopwords" indexed="true" stored="true"/>
>
> <!-- without stop words -->
> <dynamicField name="*percentmatchanyword.soundex_abc"
> type="abc_word_based_default_soundex" indexed="true" stored="true"/>
> <dynamicField name="*percentmatchanyword_nvl_abc"
> type="abc_word_based_nvl" indexed="true" stored="true"/>
> <dynamicField name="*percentmatchanyword_abc"
> type="abc_word_based_default" indexed="true" stored="true"/>
>
> <!-- Main type always stored -->
> <!-- IMPORTANT: Least specific patterns MUST BE LAST IN LIST!!! -->
> <dynamicField name="*_nvl_abc" type="abc_nvl" indexed="true"
> stored="true"/>
>
> <!-- IMPORTANT: This is the default type "percentmatchwholefield" -->
> <dynamicField name="*_abc" type="abc_default" indexed="true"
> stored="true"/>
>
>
> <!-- TODO: Eventually declare Variants NOT Stored, but stored is nice
> for debugging -->
>
>
>
> <dynamicField name="*_i" type="int" indexed="true" stored="true"/>
> <dynamicField name="*_is" type="int" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_s" type="string" indexed="true" stored="true"
> />
> <dynamicField name="*_ss" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_l" type="long" indexed="true" stored="true"/>
> <dynamicField name="*_ls" type="long" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_t" type="text_general" indexed="true"
> stored="true"/>
> <dynamicField name="*_txt" type="text_general" indexed="true"
> stored="true" multiValued="true"/>
> <dynamicField name="*_en" type="text_en" indexed="true"
> stored="true" multiValued="true"/>
> <dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
> <dynamicField name="*_bs" type="boolean" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_f" type="float" indexed="true" stored="true"/>
> <dynamicField name="*_fs" type="float" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_d" type="double" indexed="true" stored="true"/>
> <dynamicField name="*_ds" type="double" indexed="true" stored="true"
> multiValued="true"/>
>
> <!-- Type used to index the lat and lon components for the "location"
> FieldType -->
> <dynamicField name="*_coordinate" type="tdouble" indexed="true"
> stored="false" />
>
> <dynamicField name="*_dt" type="date" indexed="true"
> stored="true"/>
> <dynamicField name="*_dts" type="date" indexed="true" stored="true"
> multiValued="true"/>
> <dynamicField name="*_p" type="location" indexed="true" stored="true"/>
>
> <!-- some trie-coded dynamic fields for faster range queries -->
> <dynamicField name="*_ti" type="tint" indexed="true" stored="true"/>
> <dynamicField name="*_tl" type="tlong" indexed="true" stored="true"/>
> <dynamicField name="*_tf" type="tfloat" indexed="true" stored="true"/>
> <dynamicField name="*_td" type="tdouble" indexed="true" stored="true"/>
> <dynamicField name="*_tdt" type="tdate" indexed="true" stored="true"/>
>
> <dynamicField name="*_pi" type="pint" indexed="true"
> stored="true"/>
> <dynamicField name="*_c" type="currency" indexed="true"
> stored="true"/>
>
> <dynamicField name="ignored_*" type="ignored" multiValued="true"/>
> <dynamicField name="attr_*" type="text_general" indexed="true"
> stored="true" multiValued="true"/>
>
> <dynamicField name="random_*" type="random" />
>
> <!-- uncomment the following to ignore any fields that don't already
> match an existing
> field name or dynamic field, rather than reporting them as an
> error.
> alternately, change the type="ignored" to some other type e.g.
> "text" if you want
> unknown fields indexed and/or stored by default -->
> <!--dynamicField name="*" type="ignored" multiValued="true" /-->
>
> </fields>
>
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a
> required field
> -->
> <uniqueKey>id</uniqueKey>
>
> <!-- DEPRECATED: The defaultSearchField is consulted by various query
> parsers when
> parsing a query string that isn't explicit about the field. Machine
> (non-user)
> generated queries are best made explicit, or they can use the "df"
> request parameter
> which takes precedence over this.
> Note: Un-commenting defaultSearchField will be insufficient if your
> request handler
> in solrconfig.xml defines "df", which takes precedence. That would need
> to be removed.
> <defaultSearchField>text</defaultSearchField> -->
>
> <!-- DEPRECATED: The defaultOperator (AND|OR) is consulted by various
> query parsers
> when parsing a query string to determine if a clause of the query should
> be marked as
> required or optional, assuming the clause isn't already marked by some
> operator.
> The default is OR, which is generally assumed so it is not a good idea
> to change it
> globally here. The "q.op" request parameter takes precedence over this.
> <solrQueryParser defaultOperator="OR"/> -->
>
> <!-- copyField commands copy one field to another at the time a document
> is added to the index. It's used either to index the same field
> differently,
> or to add multiple fields to the same field for easier/faster
> searching. -->
>
> <copyField source="cat" dest="text"/>
> <copyField source="name" dest="text"/>
> <copyField source="manu" dest="text"/>
> <copyField source="features" dest="text"/>
> <copyField source="includes" dest="text"/>
> <copyField source="manu" dest="manu_exact"/>
>
> <!-- Copy the price into a currency enabled field (default USD) -->
> <copyField source="price" dest="price_c"/>
>
> <!-- Text fields from SolrCell to search by default in our catch-all
> field -->
> <copyField source="title" dest="text"/>
> <copyField source="author" dest="text"/>
> <copyField source="description" dest="text"/>
> <copyField source="keywords" dest="text"/>
> <copyField source="content" dest="text"/>
> <copyField source="content_type" dest="text"/>
> <copyField source="resourcename" dest="text"/>
> <copyField source="url" dest="text"/>
>
> <!-- Create a string version of author for faceting -->
> <copyField source="author" dest="author_s"/>
> <!-- Above, multiple source fields are copied to the [text] field.
> Another way to map multiple source fields to the same
> destination field is to use the dynamic field syntax.
> copyField also supports a maxChars to copy setting. -->
>
> <!-- <copyField source="*_t" dest="text" maxChars="3000"/> -->
>
> <!-- copy name to alphaNameSort, a field designed for sorting by name
> -->
> <!-- <copyField source="name" dest="alphaNameSort"/> -->
>
> <!-- Abc Copy Fields -->
>
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.case_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.case.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.case_nvl_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.string_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.RECORD_ID_abc"
> dest="ADDRESS_ALL.RECORD_ID_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.typo_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.string_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
> dest="ADDRESS_ALL.ADDR_TYPE_CD_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.firstnamesthes_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.firstnamesthes.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.firstnamesthes_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.typo_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.string_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.LONGITUDE_abc"
> dest="ADDRESS_ALL.LONGITUDE_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorepunctuation_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorepunctuation.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.ignorepunctuation_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.typo_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.string_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.LATITUDE_abc"
> dest="ADDRESS_ALL.LATITUDE_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.typo_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.string_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.ADDR_NAME_abc"
> dest="ADDRESS_ALL.ADDR_NAME_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.citynamesthes_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.citynamesthes.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.citynamesthes_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc" dest="ADDRESS_ALL.CITY.typo_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.string_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc"
> dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.CITY_abc" dest="ADDRESS_ALL.CITY_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.string_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly.soundex_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc"
> dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.STATE_abc" dest="ADDRESS_ALL.STATE_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.soundex_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.typo_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.typo.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.typo_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.string_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.string.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.string_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword.soundex_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly.soundex_abc"
> />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly_nvl_abc" />
> <copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
> dest="ADDRESS_ALL.EMAIL_ADDR_nvl_abc" />
>
>
>
> <types>
> <!-- field type definitions. The "name" attribute is
> just a label to be used by field definitions. The "class"
> attribute and any other attributes determine the real
> behavior of the fieldType.
> Class names starting with "solr" refer to java classes in a
> standard package such as org.apache.solr.analysis
> -->
>
> <!-- The StrField type is not analyzed, but indexed/stored verbatim.
> -->
> <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> />
>
> <!-- boolean type: "true" or "false" -->
> <fieldType name="boolean" class="solr.BoolField"
> sortMissingLast="true"/>
>
> <!-- sortMissingLast and sortMissingFirst attributes are optional
> attributes are
> currently supported on types that are sorted internally as strings
> and on numeric types.
> This includes "string","boolean", and, as of 3.5 (and 4.x),
> int, float, long, date, double, including the "Trie" variants.
> - If sortMissingLast="true", then a sort on this field will cause
> documents
> without the field to come after documents with the field,
> regardless of the requested sort order (asc or desc).
> - If sortMissingFirst="true", then a sort on this field will cause
> documents
> without the field to come before documents with the field,
> regardless of the requested sort order.
> - If sortMissingLast="false" and sortMissingFirst="false" (the
> default),
> then default lucene sorting will be used which places docs
> without the
> field first in an ascending sort and last in a descending sort.
> -->
>
> <!--
> Default numeric field types. For faster range queries, consider the
> tint/tfloat/tlong/tdouble types.
> -->
> <fieldType name="int" class="solr.TrieIntField" precisionStep="0"
> positionIncrementGap="0"/>
> <fieldType name="float" class="solr.TrieFloatField" precisionStep="0"
> positionIncrementGap="0"/>
> <fieldType name="long" class="solr.TrieLongField" precisionStep="0"
> positionIncrementGap="0"/>
> <fieldType name="double" class="solr.TrieDoubleField"
> precisionStep="0" positionIncrementGap="0"/>
>
> <!--
> Numeric field types that index each value at various levels of
> precision
> to accelerate range queries when the number of values between the
> range
> endpoints is large. See the javadoc for NumericRangeQuery for internal
> implementation details.
>
> Smaller precisionStep values (specified in bits) will lead to more
> tokens
> indexed per value, slightly larger index size, and faster range
> queries.
> A precisionStep of 0 disables indexing at different precision levels.
> -->
> <fieldType name="tint" class="solr.TrieIntField" precisionStep="8"
> positionIncrementGap="0"/>
> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8"
> positionIncrementGap="0"/>
> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8"
> positionIncrementGap="0"/>
> <fieldType name="tdouble" class="solr.TrieDoubleField"
> precisionStep="8" positionIncrementGap="0"/>
>
> <!-- The format for this date field is of the form
> 1995-12-31T23:59:59Z, and
> is a more restricted form of the canonical representation of
> dateTime
> http://www.w3.org/TR/xmlschema-2/#dateTime
> The trailing "Z" designates UTC time and is mandatory.
> Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
> All other components are mandatory.
>
> Expressions can also be used to denote calculations that should be
> performed relative to "NOW" to determine the value, ie...
>
> NOW/HOUR
> ... Round to the start of the current hour
> NOW-1DAY
> ... Exactly 1 day prior to now
> NOW/DAY+6MONTHS+3DAYS
> ... 6 months and 3 days in the future from the start of
> the current day
>
> Consult the DateField javadocs for more information.
>
> Note: For faster range queries, consider the tdate type
> -->
> <fieldType name="date" class="solr.TrieDateField" precisionStep="0"
> positionIncrementGap="0"/>
>
> <!-- A Trie based date field for faster date range queries and date
> faceting. -->
> <fieldType name="tdate" class="solr.TrieDateField" precisionStep="6"
> positionIncrementGap="0"/>
>
>
> <!--Binary data type. The data should be sent/retrieved in as Base64
> encoded Strings -->
> <fieldType name="binary" class="solr.BinaryField"/>
>
> <!--
> Note:
> These should only be used for compatibility with existing indexes
> (created with lucene or older Solr versions).
> Use Trie based fields instead. As of Solr 3.5 and 4.x, Trie based
> fields support sortMissingFirst/Last
>
> Plain numeric field types that store and index the text
> value verbatim (and hence don't correctly support range queries,
> since the
> lexicographic ordering isn't equal to the numeric ordering)
> -->
> <fieldType name="pint" class="solr.IntField"/>
> <fieldType name="plong" class="solr.LongField"/>
> <fieldType name="pfloat" class="solr.FloatField"/>
> <fieldType name="pdouble" class="solr.DoubleField"/>
> <fieldType name="pdate" class="solr.DateField" sortMissingLast="true"/>
>
> <!-- The "RandomSortField" is not used to store or search any
> data. You can declare fields of this type it in your schema
> to generate pseudo-random orderings of your docs for sorting
> or function purposes. The ordering is generated based on the
> field
> name and the version of the index. As long as the index version
> remains unchanged, and the same field name is reused,
> the ordering of the docs will be consistent.
> If you want different psuedo-random orderings of documents,
> for the same version of the index, use a dynamicField and
> change the field name in the request.
> -->
> <fieldType name="random" class="solr.RandomSortField" indexed="true" />
>
> <!-- Abc Types -->
> <!-- ========== -->
> <!-- Most general fields will be IDs
> So basing this more closely on "string"
> -->
> <!--
> <fieldType name="abc_id" class="solr.StrField" sortMissingLast="true"
> positionIncrementGap="100" />
> -->
> <fieldType name="abc_id" class="solr.TextField" sortMissingLast="true">
> <analyzer>
> <!-- Keyword doesn't actually tokenize anything -->
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_fulltext" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Glue everything left into one giant token -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
>
> <!-- If you change this, see also FirstFiveDigitsUtil and names for
> USA -->
> <fieldType name="abc_default" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Glue everything left into one giant token -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <!-- Generic Null Value / Non Null Value -->
> <!-- Some types, such as Addresses, have their own nvl types -->
> <fieldType name="abc_nvl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_word_based_default" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_word_based_default_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Generic Null Value / Non Null Value -->
> <!-- Some types, such as Addresses, have their own nvl types -->
> <fieldType name="abc_word_based_nvl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <fieldType name="abc_word_based_stopwords" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_word_based_stopwords_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Generic Null Value / Non Null Value -->
> <!-- Some types, such as Addresses, have their own nvl types -->
> <fieldType name="abc_word_based_stopwords_nvl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
>
> <fieldType name="abc_ignorecase" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_ignorepunctuation" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory" />
> <!--<filter class="solr.WordDelimiterFilterFactory" />-->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_ignorepunctuation_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory" />
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!--<filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
> inject="false" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_porterstem" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!-- <tokenizer class="solr.WhitespaceTokenizerFactory" /> -->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Need Lowercase before Snowball -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <!--<filter class="solr.PorterStemFilterFactory" />-->
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_porterstem_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_firstfivedigits" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!-- Keyword doesn't actually tokenize anything -->
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!-- discard anything that's not 5 digits
> including nuke the entire field if not starting with 5 digits
> -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="^([0-9]{5})?.*"
> replacement="$1" replace="all" />
>
>
> <!--<filter class="com.abc.fulltext.multiop.FirstFiveDigitsFilterFactory"
> />-->
> <!-- Step 1: Remove everything that's not a digit -->
> <!--<filter class="solr.PatternReplaceFilterFactory" pattern="[^0-9]"
> replacement="" replace="all" />-->
> <!-- Step 2: Keep only first 5 digits -->
> <!--<filter class="solr.PatternReplaceFilterFactory"
> pattern="^([0-9]{1,5}).*" replacement="$1" replace="all" />-->
> <!-- Step 3: Discard anything not 5 digits -->
> <!--<filter class="solr.LengthFilterFactory" min="5" max="5" />-->
>
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- No lowercase since just digits -->
> </analyzer>
> </fieldType>
> <fieldType name="abc_ignorenonalphanumeric" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!-- Keyword doesn't actually tokenize anything -->
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!--<filter
> class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
> <!-- throw out all everything except a-z and 0-9, include A-Z just to be
> safe -->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Don't need to glue back together since used Keyword -->
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_ignorenonalphanumeric_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_typo" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <!-- <tokenizer class="solr.WhitespaceTokenizerFactory" /> -->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!--<filter class="solr.PhoneticFilterFactory" encoder="Metaphone" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_porterstem_ignorenonalphanumeric"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer>
> <!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <!-- Need Lowercase before Snowball -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <!--<filter class="solr.PorterStemFilterFactory" />-->
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <!--<filter
> class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
> <!-- Throw out all everything except a-z and 0-9, include A-Z just to be
> safe -->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Glue all the tokens back together -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_porterstem_ignorenonalphanumeric_soundex"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <!-- Need Lowercase before Snowball -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <!--<filter class="solr.PorterStemFilterFactory" />-->
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <!--<filter
> class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
> <!-- throw out all everything except a-z and 0-9, include A-Z just to be
> safe -->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Glue all the tokens back together -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!--<filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
> inject="false" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_stopwords_porterstem_ignorenonalphanumeric"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer>
> <!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> <!-- Need Lowercase before Snowball -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <!--<filter class="solr.PorterStemFilterFactory" />-->
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <!--<filter
> class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
> <!-- throw out all everything except a-z and 0-9, include A-Z just to be
> safe -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
> replacement="" replace="all" />
> <!-- Glue all the tokens back together -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
> <!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> <!-- Need Lowercase before Snowball -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <!--<filter class="solr.PorterStemFilterFactory" />-->
> <filter class="solr.SnowballPorterFilterFactory" language="English" />
> <!--<filter
> class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
> <!-- Throw out all everything except a-z and 0-9, include A-Z just to be
> safe -->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-zA-Z0-9])" replacement="" replace="all" />
> <!-- Glue all the tokens back together -->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- <filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
> inject="false" /> -->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_stopwords" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_stopwords_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/stopwords.txt" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_junk" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/junk.txt" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_profanity" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="wordlists/profanity.txt" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_areacodethes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[^0-9]" />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/areacodethes.txt" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_citynamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/citynamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/citynamesthes_abbrev.txt" />-->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_citynamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/citynamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/citynamesthes_abbrev.txt" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_companynamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/companynamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/companynamesthes_abbrev.txt" />-->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_companynamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/companynamesthes_expand.txt" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_countrynamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/countrynamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/countrynamesthes_abbrev.txt" />-->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_countrynamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/countrynamesthes_expand.txt" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_firstnamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/firstnamesthes.txt" />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_firstnamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/firstnamesthes.txt" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_provincenamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/provincenamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/provincenamesthes_abbrev.txt" />-->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_provincenamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/provincenamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/provincenamesthes_abbrev.txt" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <!-- thes changed to index and query time due to concatenation -->
> <fieldType name="abc_statenamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="query">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/statenamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/statenamesthes_abbrev.txt" />-->
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_statenamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.PatternTokenizerFactory"
> pattern="[-
> \t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
> />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/statenamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/statenamesthes_abbrev.txt" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- some other thresaurus items in the address section -->
> <!--
> *********************************************
> Begin Addresses: Abc fieldTypes
> *********************************************
> streetname
> streetnumber
> streetsecondary
> streetpredirectional
> streetpostdirectional
> streetsuffix
> pobox
> matchnullone matchnullboth
> -->
>
> <!-- Not using
> <filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
> Because we're using
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> which means already only one token
> unless stop words or thes or whatever is used
> -->
>
> <!-- streetname / Street Name -->
> <fieldType name="abc_streetnamepercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamepercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_streetnamepercentage_soundex"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamepercentage" />
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <!-- Keep LowerCase last due to Soundex -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnamepercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamepercentage" />
> <!-- Only added at Index Time -->
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetnamepercentage.typo" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamepercentage.typo" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnamepercentage.typo" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamepercentage.typo" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <fieldType name="abc_streetnamematchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamematchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnamematchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamematchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetnamematchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamematchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnamematchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamematchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- TODO: plural vs. singular? abc_streetnamesthes vs. abc_streetnamethes
> -->
> <fieldType name="abc_streetnamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamesthes" />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
> <!-- TODO: consider concat filter -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
> <!-- TODO: consider concat filter -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_streetnamesthes_soundex" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamesthes" />
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/streetnamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
> <!-- Replace numbers/digits with words -->
> <filter class="solr.PatternReplaceFilterFactory" pattern="0"
> replacement="zero" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="1"
> replacement="one" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="2"
> replacement="two" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="3"
> replacement="three" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="4"
> replacement="four" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="5"
> replacement="five" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="6"
> replacement="six" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="7"
> replacement="seven" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="8"
> replacement="eight" replace="all" />
> <filter class="solr.PatternReplaceFilterFactory" pattern="9"
> replacement="nine" replace="all" />
> <filter class="solr.PhoneticFilterFactory"
> encoder="DoubleMetaphone"
> inject="false"
> maxCodeLength="25"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnamesthes" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnamesthes" />
> <!-- TODO: Break up multi word street names, etc. at this point, need
> token filter, not tokenizer -->
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!-- TODO: Break up multi word street names, etc. at this point, need
> token filter, not tokenizer -->
> <filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
> synonyms="wordlists/streetnamesthes_expand.txt" />
> <!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
> expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- streetnumber / Street Number -->
> <fieldType name="abc_streetnumberpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumberpercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnumberpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumberpercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetnumbermatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumbermatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnumbermatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumbermatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetnumbermatchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumbermatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetnumbermatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetnumbermatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- streetsecondary / Street Secondary -->
> <fieldType name="abc_streetsecondarypercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarypercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsecondarypercentage"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarypercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_streetsecondarymatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarymatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsecondarymatchnullone"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarymatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetsecondarymatchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarymatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsecondarymatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsecondarymatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- streetpredirectional / Street Predirectional / Street Pre Directional
> -->
> <fieldType name="abc_streetpredirectionalpercentage"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalpercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpredirectionalpercentage"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalpercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetpredirectionalmatchnullone"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalmatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpredirectionalmatchnullone"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalmatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetpredirectionalmatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalmatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpredirectionalmatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpredirectionalmatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- streetpostdirectional / Street Postdirectional / Street Post
> Directional -->
> <fieldType name="abc_streetpostdirectionalpercentage"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalpercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpostdirectionalpercentage"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalpercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetpostdirectionalmatchnullone"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalmatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpostdirectionalmatchnullone"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalmatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetpostdirectionalmatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalmatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetpostdirectionalmatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetpostdirectionalmatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- streetsuffix / Street Suffix -->
> <fieldType name="abc_streetsuffixpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixpercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsuffixpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixpercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetsuffixmatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixmatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsuffixmatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixmatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_streetsuffixmatchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixmatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_streetsuffixmatchnullboth"
> class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="streetsuffixmatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- pobox / PO Box -->
> <fieldType name="abc_poboxpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxpercentage" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_poboxpercentage" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxpercentage" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_poboxmatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxmatchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_poboxmatchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxmatchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="abc_poboxmatchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxmatchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_poboxmatchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="poboxmatchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <!-- matchnullone matchnullboth / Match Null One / Match Null Both -->
> <!-- TODO: not sure we still need these? Also in dynamicField section -->
> <fieldType name="abc_matchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="matchnullone" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_matchnullone" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="matchnullone" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- matchnullboth Match Null Both -->
> <fieldType name="abc_matchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="matchnullboth" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <fieldType name="abc_nvl_matchnullboth" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
> operator="matchnullboth" />
> <filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
>
>
>
>
>
>
> <!-- solr.TextField allows the specification of custom text analyzers
> specified as a tokenizer and a list of token filters. Different
> analyzers may be specified for indexing and querying.
>
> The optional positionIncrementGap puts space between multiple
> fields of
> this type on the same document, with the purpose of preventing
> false phrase
> matching across fields.
>
> For more info on customizing your analyzer chain, please see
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
> -->
>
> <!-- One can also specify an existing Analyzer class that has a
> default constructor via the class attribute on the analyzer
> element.
> Example:
> <fieldType name="text_greek" class="solr.TextField">
> <analyzer class="org.apache.lucene.analysis.el.GreekAnalyzer"/>
> </fieldType>
> -->
>
> <!-- A text field that only splits on whitespace for exact matching of
> words -->
> <fieldType name="text_ws" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- A general text field that has reasonable, generic
> cross-language defaults: it tokenizes with StandardTokenizer,
> removes stop words from case-insensitive "stopwords.txt"
> (empty by default), and down cases. At query time only, it
> also applies synonyms. -->
> <fieldType name="text_general" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true" />
> <!-- in this example, we will only use synonyms at query time
> <filter class="solr.SynonymFilterFactory"
> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true" />
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- A text field with defaults appropriate for English: it
> tokenizes with StandardTokenizer, removes English stop words
> (lang/stopwords_en.txt), down cases, protects words from
> protwords.txt, and
> finally applies Porter's stemming. The query time analyzer
> also applies synonyms from synonyms.txt. -->
> <fieldType name="text_en" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- in this example, we will only use synonyms at query time
> <filter class="solr.SynonymFilterFactory"
> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> -->
> <!-- Case insensitive stop word removal.
> add enablePositionIncrements=true in both the index and query
> analyzers to leave a 'gap' for more accurate phrase queries.
> -->
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="lang/stopwords_en.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <!-- Optionally you may want to use this less aggressive stemmer instead
> of PorterStemFilterFactory:
> <filter class="solr.EnglishMinimalStemFilterFactory"/>
> -->
> <filter class="solr.PorterStemFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="lang/stopwords_en.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <!-- Optionally you may want to use this less aggressive stemmer instead
> of PorterStemFilterFactory:
> <filter class="solr.EnglishMinimalStemFilterFactory"/>
> -->
> <filter class="solr.PorterStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- A text field with defaults appropriate for English, plus
> aggressive word-splitting and autophrase features enabled.
> This field is just like text_en, except it adds
> WordDelimiterFilter to enable splitting and matching of
> words on case-change, alpha numeric boundaries, and
> non-alphanumeric chars. This means certain compound word
> cases will work, for example query "wi fi" will match
> document "WiFi" or "wi-fi".
> -->
> <fieldType name="text_en_splitting" class="solr.TextField"
> positionIncrementGap="100" autoGeneratePhraseQueries="true">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <!-- in this example, we will only use synonyms at query time
> <filter class="solr.SynonymFilterFactory"
> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> -->
> <!-- Case insensitive stop word removal.
> add enablePositionIncrements=true in both the index and query
> analyzers to leave a 'gap' for more accurate phrase queries.
> -->
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="lang/stopwords_en.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.PorterStemFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="lang/stopwords_en.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.PorterStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Less flexible matching, but less false matches. Probably not
> ideal for product names,
> but may be good for SKUs. Can insert dashes in the wrong place
> and still match. -->
> <fieldType name="text_en_splitting_tight" class="solr.TextField"
> positionIncrementGap="100" autoGeneratePhraseQueries="true">
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="false"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_en.txt"/>
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="0" catenateWords="1"
> catenateNumbers="1" catenateAll="0"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.EnglishMinimalStemFilterFactory"/>
> <!-- this filter can remove any duplicate tokens that appear at
> the same position - sometimes
> possible with WordDelimiterFilter in conjuncton with
> stemming. -->
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Just like text_general except it reverses the characters of
> each token, to enable more efficient leading wildcard queries. -->
> <fieldType name="text_general_rev" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true" />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ReversedWildcardFilterFactory"
> withOriginal="true"
> maxPosAsterisk="3" maxPosQuestion="2"
> maxFractionAsterisk="0.33"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- charFilter + WhitespaceTokenizer -->
> <!--
> <fieldType name="text_char_norm" class="solr.TextField"
> positionIncrementGap="100" >
> <analyzer>
> <charFilter class="solr.MappingCharFilterFactory"
> mapping="mapping-ISOLatin1Accent.txt"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> </analyzer>
> </fieldType>
> -->
>
> <!-- This is an example of using the KeywordTokenizer along
> With various TokenFilterFactories to produce a sortable field
> that does not include some properties of the source text
> -->
> <fieldType name="alphaOnlySort" class="solr.TextField"
> sortMissingLast="true" omitNorms="true">
> <analyzer>
> <!-- KeywordTokenizer does no actual tokenizing, so the entire
> input string is preserved as a single token
> -->
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <!-- The LowerCase TokenFilter does what you expect, which can be
> when you want your sorting to be case insensitive
> -->
> <filter class="solr.LowerCaseFilterFactory" />
> <!-- The TrimFilter removes any leading or trailing whitespace -->
> <filter class="solr.TrimFilterFactory" />
> <!-- The PatternReplaceFilter gives you the flexibility to use
> Java Regular expression to replace any sequence of characters
> matching a pattern with an arbitrary replacement string,
> which may include back references to portions of the original
> string matched by the pattern.
>
> See the Java Regular Expression documentation for more
> information on pattern and replacement string syntax.
>
>
> http://java.sun.com/j2se/1.6.0/docs/api/java/util/regex/package-summary.html
> -->
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="([^a-z])" replacement="" replace="all"
> />
> </analyzer>
> </fieldType>
>
> <fieldType name="phonetic" stored="false" indexed="true"
> class="solr.TextField" >
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.DoubleMetaphoneFilterFactory" inject="false"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="payloads" stored="false" indexed="true"
> class="solr.TextField" >
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <!--
> The DelimitedPayloadTokenFilter can put payloads on tokens... for
> example,
> a token of "foo|1.4" would be indexed as "foo" with a payload of
> 1.4f
> Attributes of the DelimitedPayloadTokenFilterFactory :
> "delimiter" - a one character delimiter. Default is | (pipe)
> "encoder" - how to encode the following value into a playload
> float -> org.apache.lucene.analysis.payloads.FloatEncoder,
> integer -> o.a.l.a.p.IntegerEncoder
> identity -> o.a.l.a.p.IdentityEncoder
> Fully Qualified class name implementing PayloadEncoder,
> Encoder must have a no arg constructor.
> -->
> <filter class="solr.DelimitedPayloadTokenFilterFactory"
> encoder="float"/>
> </analyzer>
> </fieldType>
>
> <!-- lowercases the entire field value, keeping it as a single token.
> -->
> <fieldType name="lowercase" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory" />
> </analyzer>
> </fieldType>
>
> <!--
> Example of using PathHierarchyTokenizerFactory at index time, so
> queries for paths match documents at that path, or in descendent
> paths
> -->
> <fieldType name="descendent_path" class="solr.TextField">
> <analyzer type="index">
> <tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory" />
> </analyzer>
> </fieldType>
> <!--
> Example of using PathHierarchyTokenizerFactory at query time, so
> queries for paths match documents at that path, or in ancestor paths
> -->
> <fieldType name="ancestor_path" class="solr.TextField">
> <analyzer type="index">
> <tokenizer class="solr.KeywordTokenizerFactory" />
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
> </analyzer>
> </fieldType>
>
> <!-- since fields of this type are by default not stored or indexed,
> any data added to them will be ignored outright. -->
> <fieldType name="ignored" stored="false" indexed="false"
> multiValued="true" class="solr.StrField" />
>
> <!-- This point type indexes the coordinates as separate fields
> (subFields)
> If subFieldType is defined, it references a type, and a dynamic field
> definition is created matching *___<typename>. Alternately, if
> subFieldSuffix is defined, that is used to create the subFields.
> Example: if subFieldType="double", then the coordinates would be
> indexed in fields myloc_0___double,myloc_1___double.
> Example: if subFieldSuffix="_d" then the coordinates would be indexed
> in fields myloc_0_d,myloc_1_d
> The subFields are an implementation detail of the fieldType, and end
> users normally should not need to know about them.
> -->
> <fieldType name="point" class="solr.PointType" dimension="2"
> subFieldSuffix="_d"/>
>
> <!-- A specialized field for geospatial search. If indexed, this
> fieldType must not be multivalued. -->
> <fieldType name="location" class="solr.LatLonType"
> subFieldSuffix="_coordinate"/>
>
> <!--
> A Geohash is a compact representation of a latitude longitude pair in
> a single field.
> See http://wiki.apache.org/solr/SpatialSearch
> -->
> <fieldType name="geohash" class="solr.GeoHashField"/>
>
> <!-- Money/currency field type. See
> http://wiki.apache.org/solr/MoneyFieldType
> Parameters:
> defaultCurrency: Specifies the default currency if none
> specified. Defaults to "USD"
> precisionStep: Specifies the precisionStep for the TrieLong
> field used for the amount
> providerClass: Lets you plug in other exchange provider
> backend:
> solr.FileExchangeRateProvider is the default
> and takes one parameter:
> currencyConfig: name of an xml file holding
> exhange rates
> solr.OpenExchangeRatesOrgProvider uses rates
> from openexchangerates.org:
> ratesFileLocation: URL or path to rates JSON
> file (default latest.json on the web)
> refreshInterval: Number of minutes between
> each rates fetch (default: 1440, min: 60)
> -->
> <fieldType name="currency" class="solr.CurrencyField"
> precisionStep="8" defaultCurrency="USD" currencyConfig="currency.xml" />
>
>
>
> <!-- some examples for different languages (generally ordered by ISO
> code) -->
>
> <!-- Arabic -->
> <fieldType name="text_ar" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- for any non-arabic -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ar.txt" enablePositionIncrements="true"/>
> <!-- normalizes ﻯ to ﻱ, etc -->
> <filter class="solr.ArabicNormalizationFilterFactory"/>
> <filter class="solr.ArabicStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Bulgarian -->
> <fieldType name="text_bg" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_bg.txt" enablePositionIncrements="true"/>
> <filter class="solr.BulgarianStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Catalan -->
> <fieldType name="text_ca" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- removes l', etc -->
> <filter class="solr.ElisionFilterFactory" ignoreCase="true"
> articles="lang/contractions_ca.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ca.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Catalan"/>
> </analyzer>
> </fieldType>
>
> <!-- CJK bigram (see text_ja for a Japanese configuration using
> morphological analysis) -->
> <fieldType name="text_cjk" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- normalize width before bigram, as e.g. half-width dakuten
> combine -->
> <filter class="solr.CJKWidthFilterFactory"/>
> <!-- for any non-CJK -->
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.CJKBigramFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Czech -->
> <fieldType name="text_cz" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_cz.txt" enablePositionIncrements="true"/>
> <filter class="solr.CzechStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Danish -->
> <fieldType name="text_da" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_da.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Danish"/>
> </analyzer>
> </fieldType>
>
> <!-- German -->
> <fieldType name="text_de" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_de.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.GermanNormalizationFilterFactory"/>
> <filter class="solr.GermanLightStemFilterFactory"/>
> <!-- less aggressive: <filter
> class="solr.GermanMinimalStemFilterFactory"/> -->
> <!-- more aggressive: <filter
> class="solr.SnowballPorterFilterFactory" language="German2"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Greek -->
> <fieldType name="text_el" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- greek specific lowercase for sigma -->
> <filter class="solr.GreekLowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="false"
> words="lang/stopwords_el.txt" enablePositionIncrements="true"/>
> <filter class="solr.GreekStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Spanish -->
> <fieldType name="text_es" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_es.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SpanishLightStemFilterFactory"/>
> <!-- more aggressive: <filter
> class="solr.SnowballPorterFilterFactory" language="Spanish"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Basque -->
> <fieldType name="text_eu" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_eu.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Basque"/>
> </analyzer>
> </fieldType>
>
> <!-- Persian -->
> <fieldType name="text_fa" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <!-- for ZWNJ -->
> <charFilter class="solr.PersianCharFilterFactory"/>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ArabicNormalizationFilterFactory"/>
> <filter class="solr.PersianNormalizationFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_fa.txt" enablePositionIncrements="true"/>
> </analyzer>
> </fieldType>
>
> <!-- Finnish -->
> <fieldType name="text_fi" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_fi.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Finnish"/>
> <!-- less aggressive: <filter
> class="solr.FinnishLightStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- French -->
> <fieldType name="text_fr" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- removes l', etc -->
> <filter class="solr.ElisionFilterFactory" ignoreCase="true"
> articles="lang/contractions_fr.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_fr.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.FrenchLightStemFilterFactory"/>
> <!-- less aggressive: <filter
> class="solr.FrenchMinimalStemFilterFactory"/> -->
> <!-- more aggressive: <filter
> class="solr.SnowballPorterFilterFactory" language="French"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Irish -->
> <fieldType name="text_ga" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- removes d', etc -->
> <filter class="solr.ElisionFilterFactory" ignoreCase="true"
> articles="lang/contractions_ga.txt"/>
> <!-- removes n-, etc. position increments is intentionally false!
> -->
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/hyphenations_ga.txt" enablePositionIncrements="false"/>
> <filter class="solr.IrishLowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ga.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory" language="Irish"/>
> </analyzer>
> </fieldType>
>
> <!-- Galician -->
> <fieldType name="text_gl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_gl.txt" enablePositionIncrements="true"/>
> <filter class="solr.GalicianStemFilterFactory"/>
> <!-- less aggressive: <filter
> class="solr.GalicianMinimalStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Hindi -->
> <fieldType name="text_hi" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <!-- normalizes unicode representation -->
> <filter class="solr.IndicNormalizationFilterFactory"/>
> <!-- normalizes variation in spelling -->
> <filter class="solr.HindiNormalizationFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_hi.txt" enablePositionIncrements="true"/>
> <filter class="solr.HindiStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Hungarian -->
> <fieldType name="text_hu" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_hu.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Hungarian"/>
> <!-- less aggressive: <filter
> class="solr.HungarianLightStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Armenian -->
> <fieldType name="text_hy" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_hy.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Armenian"/>
> </analyzer>
> </fieldType>
>
> <!-- Indonesian -->
> <fieldType name="text_id" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_id.txt" enablePositionIncrements="true"/>
> <!-- for a less aggressive approach (only inflectional suffixes),
> set stemDerivational to false -->
> <filter class="solr.IndonesianStemFilterFactory"
> stemDerivational="true"/>
> </analyzer>
> </fieldType>
>
> <!-- Italian -->
> <fieldType name="text_it" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!-- removes l', etc -->
> <filter class="solr.ElisionFilterFactory" ignoreCase="true"
> articles="lang/contractions_it.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_it.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.ItalianLightStemFilterFactory"/>
> <!-- more aggressive: <filter
> class="solr.SnowballPorterFilterFactory" language="Italian"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Japanese using morphological analysis (see text_cjk for a
> configuration using bigramming)
>
> NOTE: If you want to optimize search for precision, use default
> operator AND in your query
> parser config with <solrQueryParser defaultOperator="AND"/>
> further down in this file. Use
> OR if you would like to optimize for recall (default).
> -->
> <fieldType name="text_ja" class="solr.TextField"
> positionIncrementGap="100" autoGeneratePhraseQueries="false">
> <analyzer>
> <!-- Kuromoji Japanese morphological analyzer/tokenizer
> (JapaneseTokenizer)
>
> Kuromoji has a search mode (default) that does segmentation
> useful for search. A heuristic
> is used to segment compounds into its parts and the compound
> itself is kept as synonym.
>
> Valid values for attribute mode are:
> normal: regular segmentation
> search: segmentation useful for search with synonyms
> compounds (default)
> extended: same as search mode, but unigrams unknown words
> (experimental)
>
> For some applications it might be good to use search mode for
> indexing and normal mode for
> queries to reduce recall and prevent parts of compounds from
> being matched and highlighted.
> Use <analyzer type="index"> and <analyzer type="query"> for
> this and mode normal in query.
>
> Kuromoji also has a convenient user dictionary feature that
> allows overriding the statistical
> model with your own entries for segmentation, part-of-speech
> tags and readings without a need
> to specify weights. Notice that user dictionaries have not
> been subject to extensive testing.
>
> User dictionary attributes are:
> userDictionary: user dictionary filename
> userDictionaryEncoding: user dictionary encoding (default is
> UTF-8)
>
> See lang/userdict_ja.txt for a sample user dictionary file.
>
> Punctuation characters are discarded by default. Use
> discardPunctuation="false" to keep them.
>
> See http://wiki.apache.org/solr/JapaneseLanguageSupport for
> more on Japanese language support.
> -->
> <tokenizer class="solr.JapaneseTokenizerFactory" mode="search"/>
> <!--<tokenizer class="solr.JapaneseTokenizerFactory" mode="search"
> userDictionary="lang/userdict_ja.txt"/>-->
> <!-- Reduces inflected verbs and adjectives to their
> base/dictionary forms (辞書形) -->
> <filter class="solr.JapaneseBaseFormFilterFactory"/>
> <!-- Removes tokens with certain part-of-speech tags -->
> <filter class="solr.JapanesePartOfSpeechStopFilterFactory"
> tags="lang/stoptags_ja.txt" enablePositionIncrements="true"/>
> <!-- Normalizes full-width romaji to half-width and half-width
> kana to full-width (Unicode NFKC subset) -->
> <filter class="solr.CJKWidthFilterFactory"/>
> <!-- Removes common tokens typically not useful for search, but
> have a negative effect on ranking -->
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ja.txt" enablePositionIncrements="true" />
> <!-- Normalizes common katakana spelling variations by removing
> any last long sound character (U+30FC) -->
> <filter class="solr.JapaneseKatakanaStemFilterFactory"
> minimumLength="4"/>
> <!-- Lower-cases romaji characters -->
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Latvian -->
> <fieldType name="text_lv" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_lv.txt" enablePositionIncrements="true"/>
> <filter class="solr.LatvianStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> <!-- Dutch -->
> <fieldType name="text_nl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_nl.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.StemmerOverrideFilterFactory"
> dictionary="lang/stemdict_nl.txt" ignoreCase="false"/>
> <filter class="solr.SnowballPorterFilterFactory" language="Dutch"/>
> </analyzer>
> </fieldType>
>
> <!-- Norwegian -->
> <fieldType name="text_no" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_no.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Norwegian"/>
> <!-- less aggressive: <filter
> class="solr.NorwegianLightStemFilterFactory"/> -->
> <!-- singular/plural: <filter
> class="solr.NorwegianMinimalStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Portuguese -->
> <fieldType name="text_pt" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_pt.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.PortugueseLightStemFilterFactory"/>
> <!-- less aggressive: <filter
> class="solr.PortugueseMinimalStemFilterFactory"/> -->
> <!-- more aggressive: <filter
> class="solr.SnowballPorterFilterFactory" language="Portuguese"/> -->
> <!-- most aggressive: <filter
> class="solr.PortugueseStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Romanian -->
> <fieldType name="text_ro" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ro.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Romanian"/>
> </analyzer>
> </fieldType>
>
> <!-- Russian -->
> <fieldType name="text_ru" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_ru.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Russian"/>
> <!-- less aggressive: <filter
> class="solr.RussianLightStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Swedish -->
> <fieldType name="text_sv" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_sv.txt" format="snowball"
> enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Swedish"/>
> <!-- less aggressive: <filter
> class="solr.SwedishLightStemFilterFactory"/> -->
> </analyzer>
> </fieldType>
>
> <!-- Thai -->
> <fieldType name="text_th" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ThaiWordFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="lang/stopwords_th.txt" enablePositionIncrements="true"/>
> </analyzer>
> </fieldType>
>
> <!-- Turkish -->
> <fieldType name="text_tr" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.TurkishLowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="false"
> words="lang/stopwords_tr.txt" enablePositionIncrements="true"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="Turkish"/>
> </analyzer>
> </fieldType>
>
> </types>
>
> <!-- Similarity is the scoring routine for each document vs. a query.
> A custom Similarity or SimilarityFactory may be specified here, but
> the default is fine for most applications.
> For more info: http://wiki.apache.org/solr/SchemaXml#Similarity
> -->
> <!--
> <similarity class="com.example.solr.CustomSimilarityFactory">
> <str name="paramkey">param value</str>
> </similarity>
> -->
>
> </schema>
>
>
>
> On Fri, Apr 4, 2014 at 4:59 PM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
>> error.txt below
>>
>>
>> Java Platform Detected x64
>> Java Platform Detected -XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+CreateMinidumpOnCrash
>> 2014-04-04 15:49:43.341:INFO:oejs.Server:jetty-8.1.8.v20121106
>> 2014-04-04 15:49:43.353:INFO:oejdp.ScanningAppProvider:Deployment monitor
>> D:\AbcData\V12\application server\server\indexer\example\contexts at
>> interval 0
>> 2014-04-04 15:49:43.358:INFO:oejd.DeploymentManager:Deployable added:
>> D:\AbcData\V12\application
>> server\server\indexer\example\contexts\solr-jetty-context.xml
>> 2014-04-04 15:49:43.989:INFO:oejw.StandardDescriptorProcessor:NO JSP
>> Support for /solr, did not find org.apache.jasper.servlet.JspServlet
>> Null identity service, trying login service: null
>> Finding identity service: null
>> 2014-04-04 15:49:44.011:INFO:oejsh.ContextHandler:started
>> o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
>> server\server\indexer\example/webapps/solr.war
>> 2014-04-04 15:49:44.012:INFO:oejsh.ContextHandler:started
>> o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
>> server\server\indexer\example/webapps/solr.war
>> Apr 04, 2014 3:49:44 PM org.apache.solr.servlet.SolrDispatchFilter init
>> INFO: SolrDispatchFilter.init()
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> locateSolrHome
>> INFO: JNDI not configured for solr (NoInitialContextEx)
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> locateSolrHome
>> INFO: solr home defaulted to 'solr/' (could not find system property or
>> JNDI)
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer$Initializer
>> initialize
>> INFO: looking for solr.xml: D:\AbcData\V12\application
>> server\server\indexer\example\solr\solr.xml
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer <init>
>> INFO: New CoreContainer 1879341237
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
>> INFO: Loading CoreContainer using Solr Home: 'solr/'
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
>> INFO: new SolrResourceLoader for directory: 'solr/'
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/apache-log4j-extras-1.2.17.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/jtds-1.2.5.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/log4j-1.2.17.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msbase.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/mssqlserver.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msutil.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/ojdbc6.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-api-1.7.5.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-nop-1.7.5.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/sqljdbc4.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting socketTimeout to: 0
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting urlScheme to: http://
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting connTimeout to: 0
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting maxConnectionsPerHost to: 20
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting corePoolSize to: 0
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting maximumPoolSize to: 2147483647
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting maxThreadIdleTime to: 5
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting sizeOfQueue to: -1
>> Apr 04, 2014 3:49:44 PM
>> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
>> INFO: Setting fairnessPolicy to: false
>> Apr 04, 2014 3:49:44 PM org.apache.solr.client.solrj.impl.HttpClientUtil
>> createClient
>> INFO: Creating new http client,
>> config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
>> INFO: Registering Log Listener
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer create
>> INFO: Creating SolrCore 'core0' using instanceDir: solr\core0
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
>> INFO: new SolrResourceLoader for directory: 'solr\core0\'
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig initLibs
>> INFO: Adding specified lib dirs to ClassLoader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-4.2.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-extras-4.2.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-core-0.7.2.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-dom-0.7.2.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcmail-jdk15-1.45.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcprov-jdk15-1.45.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/boilerpipe-1.1.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/commons-compress-1.4.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/dom4j-1.6.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/fontbox-1.7.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/icu4j-49.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/isoparser-1.0-RC-1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jdom-1.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jempbox-1.7.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/juniversalchardet-1.0.3.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/metadata-extractor-2.6.2.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/netcdf-4.2-min.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/pdfbox-1.7.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-3.8.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-3.8.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-schemas-3.8.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-scratchpad-3.8.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/rome-0.9.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tagsoup-1.2.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-core-1.3.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-parsers-1.3.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-core-0.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-tika-0.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xercesImpl-2.9.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xmlbeans-2.3.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xz-1.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/attributes-binder-1.0.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/carrot2-mini-3.6.2.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/hppc-0.4.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-core-asl-1.7.4.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-mapper-asl-1.7.4.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-collections-1.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-math-0.6.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/simple-xml-2.6.4.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/jsonic-1.2.7.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/langdetect-1.1-20120112.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-beanutils-1.7.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-collections-3.2.1.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-1.7.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-tools-2.0.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/common.jar' to
>> classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/webapp.jar' to
>> classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
>> replaceClassLoader
>> INFO: Adding
>> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/commons-pool-1.4.jar'
>> to classloader
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
>> INFO: Using Lucene MatchVersion: LUCENE_42
>> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
>> INFO: Loaded SolrConfig: solrconfig.xml
>> Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
>> INFO: Reading Solr Schema
>> Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
>> INFO: Schema name=example
>> Apr 04, 2014 3:49:46 PM org.apache.solr.schema.IndexSchema readSchema
>> INFO: unique key field: id
>> Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
>> reload
>> INFO: Reloading exchange rates from file currency.xml
>> Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
>> reload
>> INFO: Reloading exchange rates from file currency.xml
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
>> INFO: org.apache.solr.core.StandardDirectoryFactory
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
>> INFO: [core0] Opening new SolrCore at solr\core0\,
>> dataDir=D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
>> INFO: JMX monitoring not detected for core: core0
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
>> INFO: [core0] Added SolrEventListener for newSearcher:
>> org.apache.solr.core.QuerySenderListener{queries=[]}
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
>> INFO: [core0] Added SolrEventListener for firstSearcher:
>> org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher
>> warming in solrconfig.xml}]}
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
>> INFO: org.apache.solr.core.StandardDirectoryFactory
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
>> INFO: return new directory for D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data forceNew: false
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore getNewIndexDir
>> INFO: New index directory detected: old=null
>> new=D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index/
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initIndex
>> WARNING: [core0] Solr index directory 'D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index' doesn't exist.
>> Creating new index...
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
>> INFO: return new directory for D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index forceNew: false
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy onCommit
>> INFO: SolrDeletionPolicy.onCommit: commits:num=1
>> commit{dir=D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1[segments_1]
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
>> INFO: created json: solr.JSONResponseWriter
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
>> INFO: adding lazy queryResponseWriter: solr.VelocityResponseWriter
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
>> INFO: created velocity: solr.VelocityResponseWriter
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
>> INFO: created xslt: solr.XSLTResponseWriter
>> Apr 04, 2014 3:49:47 PM org.apache.solr.response.XSLTResponseWriter init
>> INFO: xsltCacheLifetimeSeconds=5
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /dataimport:
>> org.apache.solr.handler.dataimport.DataImportHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /select: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /query: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /browse: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /update: solr.XmlUpdateRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.extraction.ExtractingRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /update/extract: solr.extraction.ExtractingRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.FieldAnalysisRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /analysis/field: solr.FieldAnalysisRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /analysis/document: solr.DocumentAnalysisRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /admin/: solr.admin.AdminHandlers
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /admin/ping: solr.PingRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /debug/dump: solr.DumpRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.ReplicationHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /replication: solr.ReplicationHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /spell: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /tvrh: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /terms: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: adding lazy requestHandler: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
>> initHandlersFromConfig
>> INFO: created /elevate: solr.SearchHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.handler.loader.XMLLoader init
>> INFO: xsltCacheLifetimeSeconds=60
>> Apr 04, 2014 3:49:47 PM org.apache.solr.handler.XmlUpdateRequestHandler
>> init
>> WARNING: Using deprecated class: XmlUpdateRequestHandler -- replace with
>> UpdateRequestHandler
>> Apr 04, 2014 3:49:47 PM org.apache.solr.search.SolrIndexSearcher <init>
>> INFO: Opening Searcher@35f79e13 main
>> Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
>> INFO: Hard AutoCommit: disabled
>> Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
>> INFO: Soft AutoCommit: disabled
>> Apr 04, 2014 3:49:47 PM
>> org.apache.solr.handler.component.SpellCheckComponent inform
>> INFO: Initializing spell checkers
>> Apr 04, 2014 3:49:47 PM
>> org.apache.solr.handler.component.SpellCheckComponent inform
>> INFO: No queryConverter defined, using default converter
>> Apr 04, 2014 3:49:47 PM
>> org.apache.solr.handler.component.QueryElevationComponent inform
>> INFO: Loading QueryElevation from: D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\conf\elevate.xml
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CoreContainer registerCore
>> INFO: registering core: core0
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener
>> newSearcher
>> INFO: QuerySenderListener sending requests to Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
>> Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
>> INFO: user.dir=D:\AbcData\V12\application server\server\indexer\example
>> Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
>> INFO: SolrDispatchFilter.init() done
>> 2014-04-04 15:49:47.722:INFO:oejs.AbstractConnector:Started
>> SocketConnector@0.0.0.0:8983
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore execute
>> INFO: [core0] webapp=null path=null
>> params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false}
>> hits=0 status=0 QTime=28
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener
>> newSearcher
>> INFO: QuerySenderListener done.
>> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore registerSearcher
>> INFO: [core0] Registered new searcher Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
>> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
>> maybeReloadConfiguration
>> INFO: Loading DIH Configuration: database.xml
>> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
>> loadDataConfig
>> INFO: Data Configuration loaded successfully
>> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
>> doFullImport
>> INFO: Starting Full Import
>> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrCore execute
>> INFO: [core0] webapp=/solr path=/dataimport
>> params={commit=true&command=full-import} status=0 QTime=12
>> Apr 04, 2014 3:50:00 PM
>> org.apache.solr.handler.dataimport.SimplePropertiesWriter
>> readIndexerProperties
>> INFO: Read dataimport.properties
>> Apr 04, 2014 3:50:00 PM org.apache.solr.update.DirectUpdateHandler2
>> deleteAll
>> INFO: [core0] REMOVING ALL DOCUMENTS FROM INDEX
>> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy onInit
>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>> commit{dir=D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
>> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1[segments_1]
>> Apr 04, 2014 3:50:00 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
>> INFO: Creating a connection for entity full-index with URL:
>> jdbc:oracle:thin:@test.abcdata.com:1521:ORCL
>> Apr 04, 2014 3:50:00 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
>> INFO: Time taken for getConnection(): 702
>> MDFilter.class:::Message inserted for testing synch between compilation
>> and running application (Roberto Corona: 05/31/2013 4:02pm
>> parsepool.getNumActive(): 0
>> Parsepool.getNumIdle(): 0
>> creating new mdParse object
>> Apr 04, 2014 3:50:17 PM org.apache.solr.core.SolrCore execute
>> INFO: [core0] webapp=/solr path=/dataimport
>> params={commit=true&command=full-import} status=0 QTime=0
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid97000.hprof ...
>> Heap dump file created [62540489 bytes in 0.169 secs]
>> Apr 04, 2014 3:50:26 PM
>> org.apache.solr.update.processor.LogUpdateProcessor finish
>> INFO: [core0] webapp=/solr path=/dataimport
>> params={commit=true&command=full-import} status=0 QTime=12
>> {deleteByQuery=*:*,add=[1-5S, 1-8G, 1+DH+740, 1+DH+741, 1+DH+744, 1+DH+745,
>> 1+DH+747, 1+DH+749, 1+DH+752, 1+DH+753, ... (4514 adds)]} 0 12
>> Apr 04, 2014 3:50:26 PM org.apache.solr.common.SolrException log
>> SEVERE: Full Import failed:java.lang.RuntimeException:
>> java.lang.RuntimeException:
>> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:266)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
>> Caused by: java.lang.RuntimeException:
>> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:406)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)
>> ... 3 more
>> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
>> ... 5 more
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator.getByteBlock(ByteBlockPool.java:103)
>> at
>> org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:202)
>> at
>> org.apache.lucene.util.ByteBlockPool.allocSlice(ByteBlockPool.java:257)
>> at
>> org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:252)
>> at
>> org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:273)
>> at
>> org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:145)
>> at
>> org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:227)
>> at
>> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:235)
>> at
>> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:165)
>> at
>> org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:254)
>> at
>> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:256)
>> at
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
>> at
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
>> at
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:201)
>> at
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:477)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
>> at
>> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
>> at
>> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
>> at
>> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:235)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:500)
>> ... 6 more
>>
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2
>> rollback
>> INFO: start rollback{}
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
>> newIndexWriter
>> INFO: Creating new IndexWriter...
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
>> newIndexWriter
>> INFO: Waiting until IndexWriter is unused... core=core0
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
>> newIndexWriter
>> INFO: Rollback old IndexWriter... core=core0
>> Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy onInit
>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>> commit{dir=D:\AbcData\V12\application
>> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
>> Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1[segments_1]
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
>> newIndexWriter
>> INFO: New IndexWriter is ready to be used.
>> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2
>> rollback
>> INFO: end_rollback
>>
>>
>>
>> On Fri, Apr 4, 2014 at 4:59 PM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>>> Guessing that the attachments won't work, I am pasting one file in each
>>> of four separate emails.
>>>
>>> database.xml
>>>
>>>
>>> <dataConfig>
>>> <dataSource
>>> name="org_only"
>>> type="JdbcDataSource"
>>> driver="oracle.jdbc.OracleDriver"
>>> url="jdbc:oracle:thin:@test.abcdata.com:1521:ORCL"
>>> user="admin"
>>> password="admin"
>>> readOnly="false"
>>> />
>>> <document>
>>>
>>>
>>> <entity name="full-index" query="
>>> select
>>>
>>> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(100)), 'null')
>>> as SOLR_ID,
>>>
>>> 'ORACLE.ADDRESS_ALL'
>>> as SOLR_CATEGORY,
>>>
>>> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(255)), ' ') as
>>> ADDRESSALLROWID,
>>> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>>> ADDRESSALLADDRTYPECD,
>>> NVL(cast(ORACLE.ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
>>> ADDRESSALLLONGITUDE,
>>> NVL(cast(ORACLE.ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
>>> ADDRESSALLLATITUDE,
>>> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
>>> ADDRESSALLADDRNAME,
>>> NVL(cast(ORACLE.ADDRESS_ALL.CITY as varchar2(255)), ' ') as
>>> ADDRESSALLCITY,
>>> NVL(cast(ORACLE.ADDRESS_ALL.STATE as varchar2(255)), ' ') as
>>> ADDRESSALLSTATE,
>>> NVL(cast(ORACLE.ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>>> ADDRESSALLEMAILADDR
>>>
>>> from ORACLE.ADDRESS_ALL
>>> " >
>>>
>>> <field column="SOLR_ID" name="id" />
>>> <field column="SOLR_CATEGORY" name="category" />
>>> <field column="ADDRESSALLROWID" name="ADDRESS_ALL.RECORD_ID_abc" />
>>> <field column="ADDRESSALLADDRTYPECD" name="ADDRESS_ALL.ADDR_TYPE_CD_abc"
>>> />
>>> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ALL.LONGITUDE_abc" />
>>> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ALL.LATITUDE_abc" />
>>> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ALL.ADDR_NAME_abc" />
>>> <field column="ADDRESSALLCITY" name="ADDRESS_ALL.CITY_abc" />
>>> <field column="ADDRESSALLSTATE" name="ADDRESS_ALL.STATE_abc" />
>>> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ALL.EMAIL_ADDR_abc" />
>>>
>>> </entity>
>>>
>>>
>>>
>>> <!-- Varaibles -->
>>> <!-- '${dataimporter.last_index_time}' -->
>>> </document>
>>> </dataConfig>
>>>
>>>
>>>
>>> On Fri, Apr 4, 2014 at 4:57 PM, Candygram For Mongo <
>>> candygram.for.mongo@gmail.com> wrote:
>>>
>>>> Does this user list allow attachments? I have four files attached
>>>> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
>>>> process again using the parameters you suggested, but not to a csv file.
>>>> It errored out quickly. We are working on the csv file run.
>>>>
>>>> Removed both <autoCommit> and <autoSoftCommit> parts/definitions from
>>>> solrconfig.xml
>>>>
>>>> Disabled tlog by removing
>>>>
>>>> <updateLog>
>>>> <str name="dir">${solr.ulog.dir:}</str>
>>>> </updateLog>
>>>>
>>>> from solrconfig.xml
>>>>
>>>> Used commit=true parameter. ?commit=true&command=full-import
>>>>
>>>>
>>>> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This may not solve your problem but generally it is recommended to
>>>>> disable auto commit and transaction logs for bulk indexing.
>>>>> And issue one commit at the very end. Do you tlogs enabled? I see
>>>>> "commit failed" in the error message thats why I am offering this.
>>>>>
>>>>> And regarding comma separated values, with this approach you focus on
>>>>> just solr importing process. You separate data acquisition phrase. And it
>>>>> is very fast load even big csv files
>>>>> http://wiki.apache.org/solr/UpdateCSV
>>>>> I have never experienced OOM during indexing, I suspect data
>>>>> acquisition has role in it.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
>>>>> candygram.for.mongo@gmail.com> wrote:
>>>>>
>>>>> We would be happy to try that. That sounds counter intuitive for the
>>>>> high volume of records we have. Can you help me understand how that might
>>>>> solve our problem?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>> >
>>>>> >Can you remove auto commit for bulk import. Commit at the very end?
>>>>> >
>>>>> >Ahmet
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>>>>> candygram.for.mongo@gmail.com> wrote:
>>>>> >In case the attached database.xml file didn't show up, I have pasted
>>>>> in the
>>>>> >contents below:
>>>>> >
>>>>> ><dataConfig>
>>>>> ><dataSource
>>>>> >name="org_only"
>>>>> >type="JdbcDataSource"
>>>>> >driver="oracle.jdbc.OracleDriver"
>>>>> >url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>>>>> >user="admin"
>>>>> >password="admin"
>>>>> >readOnly="false"
>>>>> >batchSize="100"
>>>>> >/>
>>>>> ><document>
>>>>> >
>>>>> >
>>>>> ><entity name="full-index" query="
>>>>> >select
>>>>> >
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>>>>> >as SOLR_ID,
>>>>> >
>>>>> >'ORCL.ADDRESS_ACCT_ALL'
>>>>> >as SOLR_CATEGORY,
>>>>> >
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>>>>> >ADDRESSALLROWID,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>>>>> >ADDRESSALLADDRTYPECD,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>>>>> >ADDRESSALLLONGITUDE,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>>>>> >ADDRESSALLLATITUDE,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>>>>> >ADDRESSALLADDRNAME,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>>>>> >ADDRESSALLCITY,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>>>>> >ADDRESSALLSTATE,
>>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>>>>> >ADDRESSALLEMAILADDR
>>>>> >
>>>>> >from ORCL.ADDRESS_ACCT_ALL
>>>>> >" >
>>>>> >
>>>>> ><field column="SOLR_ID" name="id" />
>>>>> ><field column="SOLR_CATEGORY" name="category" />
>>>>> ><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc"
>>>>> />
>>>>> ><field column="ADDRESSALLADDRTYPECD"
>>>>> >name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>>>>> ><field column="ADDRESSALLLONGITUDE"
>>>>> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
>>>>> ><field column="ADDRESSALLLATITUDE"
>>>>> name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
>>>>> ><field column="ADDRESSALLADDRNAME"
>>>>> name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
>>>>> ><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>>>>> ><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>>>>> ><field column="ADDRESSALLEMAILADDR"
>>>>> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>>>>> >/>
>>>>> >
>>>>> ></entity>
>>>>> >
>>>>> >
>>>>> >
>>>>> ><!-- Varaibles -->
>>>>> ><!-- '${dataimporter.last_index_time}' -->
>>>>> ></document>
>>>>> ></dataConfig>
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>>>>> >candygram.for.mongo@gmail.com> wrote:
>>>>> >
>>>>> >> In this case we are indexing an Oracle database.
>>>>> >>
>>>>> >> We do not include the data-config.xml in our distribution. We
>>>>> store the
>>>>> >> database information in the database.xml file. I have attached the
>>>>> >> database.xml file.
>>>>> >>
>>>>> >> When we use the default merge policy settings, we get the same
>>>>> results.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> We have not tried to dump the table to a comma separated file. We
>>>>> think
>>>>> >> that dumping this size table to disk will introduce other memory
>>>>> problems
>>>>> >> with big file management. We have not tested that case.
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
>>>>> wrote:
>>>>> >>
>>>>> >>> Hi,
>>>>> >>>
>>>>> >>> Which database are you using? Can you send us data-config.xml?
>>>>> >>>
>>>>> >>> What happens when you use default merge policy settings?
>>>>> >>>
>>>>> >>> What happens when you dump your table to Comma Separated File and
>>>>> fed
>>>>> >>> that file to solr?
>>>>> >>>
>>>>> >>> Ahmet
>>>>> >>>
>>>>> >>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>>>>> >>> candygram.for.mongo@gmail.com> wrote:
>>>>> >>>
>>>>> >>> The ramBufferSizeMB was set to 6MB only on the test system to make
>>>>> the
>>>>> >>> system crash sooner. In production that tag is commented out which
>>>>> >>> I believe forces the default value to be used.
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi,
>>>>> >>> >
>>>>> >>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>>>>> >>> >
>>>>> >>> >Ahmet
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>>>>> >>> candygram.for.mongo@gmail.com> wrote:
>>>>> >>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>>>>> Exception
>>>>> >>> >
>>>>> >>> >*SOLR/Lucene version: *4.2.1*
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >*JVM version:
>>>>> >>> >
>>>>> >>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>>>>> >>> >
>>>>> >>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >*Indexer startup command:
>>>>> >>> >
>>>>> >>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >java " %JVMARGS% ^
>>>>> >>> >
>>>>> >>> >-Dcom.sun.management.jmxremote.port=1092 ^
>>>>> >>> >
>>>>> >>> >-Dcom.sun.management.jmxremote.ssl=false ^
>>>>> >>> >
>>>>> >>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>>>>> >>> >
>>>>> >>> >-jar start.jar
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >*SOLR indexing HTTP parameters request:
>>>>> >>> >
>>>>> >>> >webapp=/solr path=/dataimport
>>>>> >>> >params={clean=false&command=full-import&wt=javabin&version=2}
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >We are getting a Java heap OOM exception when indexing (updating)
>>>>> 27
>>>>> >>> >million records. If we increase the Java heap memory settings the
>>>>> >>> problem
>>>>> >>> >goes away but we believe the problem has not been fixed and that
>>>>> we will
>>>>> >>> >eventually get the same OOM exception. We have other processes
>>>>> on the
>>>>> >>> >server that also require resources so we cannot continually
>>>>> increase the
>>>>> >>> >memory settings to resolve the OOM issue. We are trying to find
>>>>> a way to
>>>>> >>> >configure the SOLR instance to reduce or preferably eliminate the
>>>>> >>> >possibility of an OOM exception.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >We can reproduce the problem on a test machine. We set the Java
>>>>> heap
>>>>> >>> >memory size to 64MB to accelerate the exception. If we increase
>>>>> this
>>>>> >>> >setting the same problems occurs, just hours later. In the test
>>>>> >>> >environment, we are using the following parameters:
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Normally we use the default solrconfig.xml file with only the
>>>>> following
>>>>> >>> jar
>>>>> >>> >file references added:
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> ><lib path="../../../../default/lib/common.jar" />
>>>>> >>> >
>>>>> >>> ><lib path="../../../../default/lib/webapp.jar" />
>>>>> >>> >
>>>>> >>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Using these values and trying to index 6 million records from the
>>>>> >>> database,
>>>>> >>> >the Java Heap Out of Memory exception is thrown very quickly.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >We were able to complete a successful indexing by further
>>>>> modifying the
>>>>> >>> >solrconfig.xml and removing all or all but one <copyfield> tags
>>>>> from the
>>>>> >>> >schema.xml file.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >The following solrconfig.xml values were modified:
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>>>> >>> >
>>>>> >>> ><int name="maxMergeAtOnce">2</int>
>>>>> >>> >
>>>>> >>> ><int name="maxMergeAtOnceExplicit">2</int>
>>>>> >>> >
>>>>> >>> ><int name="segmentsPerTier">10</int>
>>>>> >>> >
>>>>> >>> ><int name="maxMergedSegmentMB">150</int>
>>>>> >>> >
>>>>> >>> ></mergePolicy>
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> ><autoCommit>
>>>>> >>> >
>>>>> >>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before
>>>>> this -- >
>>>>> >>> >
>>>>> >>> ><openSearcher>false</openSearcher>
>>>>> >>> >
>>>>> >>> ></autoCommit>
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Using our customized schema.xml file with two or more <copyfield>
>>>>> tags,
>>>>> >>> the
>>>>> >>> >OOM exception is always thrown. Based on the errors, the problem
>>>>> occurs
>>>>> >>> >when the process was trying to do the merge. The error is
>>>>> provided
>>>>> >>> below:
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Exception in thread "Lucene Merge Thread #156"
>>>>> >>> >org.apache.lucene.index.MergePolicy$MergeException:
>>>>> >>> >java.lang.OutOfMemoryError: Java heap space
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>>>> >>> >
>>>>> >>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>>>> >>> >
>>>>> >>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>>>>> >>> >
>>>>> >>> >SEVERE: auto commit error...:java.lang.IllegalStateException:
>>>>> this writer
>>>>> >>> >hit an OutOfMemoryError; cannot commit
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> >>> >
>>>>> >>> > at
>>>>> >>>
>>>>> >>>
>>>>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>> >>> >
>>>>> >>> > at java.lang.Thread.run(Thread.java:722)
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >We think but are not 100% sure that the problem is related to the
>>>>> merge.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Normally our schema.xml contains a lot of field specifications
>>>>> (like the
>>>>> >>> >ones seen in the file fragment below):
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>>> >>> dest="ADDRESS.RECORD_ID.case_abc"
>>>>> >>> >/>
>>>>> >>> >
>>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>>> >>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>>>>> >>> >
>>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>>> >>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >In tests using the default file schema.xml and no <copyfield>
>>>>> tags,
>>>>> >>> >indexing completed successfully. 6 million records produced a
>>>>> 900 MB
>>>>> >>> data
>>>>> >>> >directory.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >When I included just one <copyfield> tag, indexing completed
>>>>> >>> successfully. 6
>>>>> >>> >million records produced a 990 MB data directory (90 MB bigger).
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >When I included just two <copyfield> tags, the index crashed with
>>>>> an OOM
>>>>> >>> >exception.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
>>>>> postponed
>>>>> >>> the
>>>>> >>> >crash.
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >The net of our test results I as follows:
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >*solrconfig.xml*
>>>>> >>> >
>>>>> >>> >*schema.xml*
>>>>> >>> >
>>>>> >>> >*result*
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >default plus only jar references
>>>>> >>> >
>>>>> >>> >default (no copyfield tags)
>>>>> >>> >
>>>>> >>> >success
>>>>> >>> >
>>>>> >>> >default plus only jar references
>>>>> >>> >
>>>>> >>> >modified with one copyfield tag
>>>>> >>> >
>>>>> >>> >success
>>>>> >>> >
>>>>> >>> >default plus only jar references
>>>>> >>> >
>>>>> >>> >modified with two copyfield tags
>>>>> >>> >
>>>>> >>> >crash
>>>>> >>> >
>>>>> >>> >additional modified settings
>>>>> >>> >
>>>>> >>> >default (no copyfield tags)
>>>>> >>> >
>>>>> >>> >success
>>>>> >>> >
>>>>> >>> >additional modified settings
>>>>> >>> >
>>>>> >>> >modified with one copyfield tag
>>>>> >>> >
>>>>> >>> >success
>>>>> >>> >
>>>>> >>> >additional modified settings
>>>>> >>> >
>>>>> >>> >modified with two copyfield tags
>>>>> >>> >
>>>>> >>> >crash
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >Our question is, what can we do to eliminate these OOM exceptions?
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
schema.xml below
<?xml version="1.0" encoding="UTF-8" ?>
<!-- Abc / Def Note:
Look for Abc tag to see changes we've made.
Main sections/tags you'll want:
<fields>
<dynamicField ... >
<uniqueKey>id</uniqueKey>
<copyField ... >
<types>
-->
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
This is the Solr schema file. This file should be named "schema.xml" and
should be in the conf directory under the solr home
(i.e. ./solr/conf/schema.xml by default)
or located where the classloader for the Solr webapp can find it.
This example schema is the recommended starting point for users.
It should be kept correct and concise, usable out-of-the-box.
For more information, on how to customize this file, please see
http://wiki.apache.org/solr/SchemaXml
PERFORMANCE NOTE: this schema includes many optional features and should
not
be used for benchmarking. To improve performance one could
- set stored="false" for all fields possible (esp large fields) when you
only need to search on the field but don't need to return the original
value.
- set indexed="false" if you don't need to search on the field, but only
return the field as a result of searching on other indexed fields.
- remove all unneeded copyField statements
- for best index size and searching performance, set "index" to false
for all general text fields, use copyField to copy them to the
catchall "text" field, and use that for searching.
- For maximum indexing performance, use the StreamingUpdateSolrServer
java client.
- Remember to run the JVM in server mode, and use a higher logging level
that avoids logging every request
-->
<schema name="example" version="1.5">
<!-- attribute "name" is the name of this schema and is only used for
display purposes.
version="x.y" is Solr's version number for the schema syntax and
semantics. It should
not normally be changed by applications.
1.0: multiValued attribute did not exist, all fields are multiValued
by nature
1.1: multiValued attribute introduced, false by default
1.2: omitTermFreqAndPositions attribute introduced, true by default
except for text fields.
1.3: removed optional field compress feature
1.4: default auto-phrase (QueryParser feature) to off
1.5: omitNorms defaults to true for primitive field types (int,
float, boolean, string...)
-->
<fields>
<!-- Valid attributes for fields:
name: mandatory - the name for the field
type: mandatory - the name of a field type from the
<types> fieldType section
indexed: true if this field should be indexed (searchable or sortable)
stored: true if this field should be retrievable
multiValued: true if this field may contain multiple values per
document
omitNorms: (expert) set to true to omit the norms associated with
this field (this disables length normalization and index-time
boosting for the field, and saves some memory). Only full-text
fields or fields that need an index-time boost need norms.
Norms are omitted for primitive (non-analyzed) types by default.
termVectors: [false] set to true to store the term vector for a
given field.
When using MoreLikeThis, fields used for similarity should be
stored for best performance.
termPositions: Store position information with the term vector.
This will increase storage costs.
termOffsets: Store offset information with the term vector. This
will increase storage costs.
required: The field is required. It will throw an error if the
value does not exist
default: a value that should be used if no value is specified
when adding a document.
-->
<!-- field names should consist of alphanumeric or underscore characters
only and
not start with a digit. This is not currently strictly enforced,
but other field names will not have first class support from all
components
and back compatibility is not guaranteed. Names with both leading and
trailing underscores (e.g. _version_) are reserved.
-->
<field name="id" type="string" indexed="true" stored="true"
required="true" multiValued="false" />
<field name="sku" type="text_en_splitting_tight" indexed="true"
stored="true" omitNorms="true"/>
<field name="name" type="text_general" indexed="true" stored="true"/>
<field name="manu" type="text_general" indexed="true" stored="true"
omitNorms="true"/>
<field name="cat" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="features" type="text_general" indexed="true" stored="true"
multiValued="true"/>
<field name="includes" type="text_general" indexed="true" stored="true"
termVectors="true" termPositions="true" termOffsets="true" />
<field name="weight" type="float" indexed="true" stored="true"/>
<field name="price" type="float" indexed="true" stored="true"/>
<field name="popularity" type="int" indexed="true" stored="true" />
<field name="inStock" type="boolean" indexed="true" stored="true" />
<field name="store" type="location" indexed="true" stored="true"/>
<!-- Common metadata fields, named specifically to match up with
SolrCell metadata when parsing rich documents such as Word, PDF.
Some fields are multiValued only because Tika currently may return
multiple values for them. Some metadata is parsed from the documents,
but there are some which come from the client context:
"content_type": From the HTTP headers of incoming stream
"resourcename": From SolrCell request param resource.name
-->
<field name="title" type="text_general" indexed="true" stored="true"
multiValued="true"/>
<field name="subject" type="text_general" indexed="true" stored="true"/>
<field name="description" type="text_general" indexed="true"
stored="true"/>
<field name="comments" type="text_general" indexed="true" stored="true"/>
<field name="author" type="text_general" indexed="true" stored="true"/>
<field name="keywords" type="text_general" indexed="true" stored="true"/>
<field name="category" type="text_general" indexed="true" stored="true"/>
<field name="resourcename" type="text_general" indexed="true"
stored="true"/>
<field name="url" type="text_general" indexed="true" stored="true"/>
<field name="content_type" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="last_modified" type="date" indexed="true" stored="true"/>
<field name="links" type="string" indexed="true" stored="true"
multiValued="true"/>
<!-- Main body of document extracted by SolrCell.
NOTE: This field is not indexed by default, since it is also copied
to "text"
using copyField below. This is to save space. Use this field for
returning and
highlighting document content. Use the "text" field to search the
content. -->
<field name="content" type="text_general" indexed="false" stored="true"
multiValued="true"/>
<!-- catchall field, containing all other searchable text fields
(implemented
via copyField further on in this schema -->
<field name="text" type="text_general" indexed="true" stored="false"
multiValued="true"/>
<!-- catchall text field that indexes tokens both normally and in
reverse for efficient
leading wildcard queries. -->
<field name="text_rev" type="text_general_rev" indexed="true"
stored="false" multiValued="true"/>
<!-- non-tokenized version of manufacturer to make it easier to sort or
group
results by manufacturer. copied from "manu" via copyField -->
<field name="manu_exact" type="string" indexed="true" stored="false"/>
<field name="payloads" type="payloads" indexed="true" stored="true"/>
<field name="_version_" type="long" indexed="true" stored="true"/>
<!-- Uncommenting the following will create a "timestamp" field using
a default value of "NOW" to indicate when each document was indexed.
-->
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
<!-- Dynamic field definitions allow using convention over configuration
for fields via the specification of patterns to match field names.
EXAMPLE: name="*_i" will match any field ending in _i (like myid_i,
z_i)
RESTRICTION: the glob-like pattern in the name attribute must have
a "*" only at the start or the end. -->
<!-- Abc Automatic Fields -->
<!-- ===================== -->
<!-- Variants Stored -->
<!-- stopwords renamed to percentmatchwholefieldcorewordsonly -->
<dynamicField
name="*percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
type="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
indexed="true" stored="true"
multiValued="true"
/>
<!--<dynamicField
name="*stopwords.porterstem.ignorenonalphanumeric.soundex_abc"
type="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
indexed="true" stored="true"/>-->
<dynamicField
name="*percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
type="abc_stopwords_porterstem_ignorenonalphanumeric"
indexed="true" stored="false"
/>
<!--<dynamicField name="*stopwords.porterstem.ignorenonalphanumeric_abc"
type="abc_stopwords_porterstem_ignorenonalphanumeric" indexed="true"
stored="false"/>-->
<dynamicField name="*porterstem.ignorenonalphanumeric.soundex_abc"
type="abc_porterstem_ignorenonalphanumeric_soundex" indexed="true"
stored="false" multiValued="true" />
<dynamicField name="*porterstem.ignorenonalphanumeric_abc"
type="abc_porterstem_ignorenonalphanumeric" indexed="true" stored="false"/>
<dynamicField name="*ignorecase_abc" type="abc_ignorecase"
indexed="true" stored="false"/>
<dynamicField name="*ignorepunctuation.soundex_abc"
type="abc_ignorepunctuation_soundex" indexed="true" stored="false"/>
<dynamicField name="*ignorepunctuation_abc" type="abc_ignorepunctuation"
indexed="true" stored="false"/>
<dynamicField name="*soundex_abc" type="abc_soundex" indexed="true"
stored="false" multiValued="true" />
<dynamicField name="*firstfivedigits_abc" type="abc_firstfivedigits"
indexed="true" stored="false"/>
<dynamicField name="*ignorenonalphanumeric.soundex_abc"
type="abc_ignorenonalphanumeric_soundex" indexed="true" stored="false"/>
<dynamicField name="*ignorenonalphanumeric_abc"
type="abc_ignorenonalphanumeric" indexed="true" stored="false"/>
<dynamicField name="*typo_abc" type="abc_typo" indexed="true"
stored="false"/>
<dynamicField name="*porterstem.soundex_abc"
type="abc_porterstem_soundex" indexed="true" stored="false"/>
<dynamicField name="*porterstem_abc" type="abc_porterstem"
indexed="true" stored="false"/>
<!-- Abc stopwords dynamicField -->
<dynamicField name="*stopwords_abc" type="abc_stopwords" indexed="true"
stored="false"/>
<dynamicField name="*percentmatchwholefieldcorewordsonly.soundex_abc"
type="abc_stopwords_soundex" indexed="true" stored="false"/>
<dynamicField name="*percentmatchwholefieldcorewordsonly_abc"
type="abc_stopwords" indexed="true" stored="false"/>
<dynamicField name="*junk_abc" type="abc_junk" indexed="true"
stored="false"/>
<dynamicField name="*profanity_abc" type="abc_profanity" indexed="true"
stored="false"/>
<!-- Abc Thesaurus dynamicFields, stored -->
<dynamicField name="*areacodethes_abc" type="abc_areacodethes"
indexed="true" stored="false"/>
<dynamicField name="*companynamesthes.soundex_abc"
type="abc_companynamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*companynamesthes_abc" type="abc_companynamesthes"
indexed="true" stored="false"/>
<dynamicField name="*firstnamesthes.soundex_abc"
type="abc_firstnamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*firstnamesthes_abc" type="abc_firstnamesthes"
indexed="true" stored="false"/>
<!-- Technically these are "address related" -->
<dynamicField name="*citynamesthes.soundex_abc"
type="abc_citynamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*citynamesthes_abc" type="abc_citynamesthes"
indexed="true" stored="false"/>
<dynamicField name="*statenamesthes.soundex_abc"
type="abc_statenamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*statenamesthes_abc" type="abc_statenamesthes"
indexed="true" stored="false"/>
<dynamicField name="*provincenamesthes.soundex_abc"
type="abc_provincenamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*provincenamesthes_abc" type="abc_provincenamesthes"
indexed="true" stored="false"/>
<dynamicField name="*countrynamesthes.soundex_abc"
type="abc_countrynamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*countrynamesthes_abc" type="abc_countrynamesthes"
indexed="true" stored="false"/>
<!-- Also streetnamethes defined with Address fields -->
<!--
*********************************************
Addresses: dynamicFields
*********************************************
streetname
streetnumber
streetsecondary
streetpredirectional
streetpostdirectional
streetsuffix
pobox
TODO: ? matchnullone matchnullboth
-->
<!-- streetname / Street Name -->
<dynamicField name="*streetnamepercentage.soundex_abc"
type="abc_streetnamepercentage_soundex" indexed="true" stored="false"/>
<dynamicField name="*streetnamepercentage_abc"
type="abc_streetnamepercentage" indexed="true" stored="false"/>
<dynamicField name="*streetnamepercentage_nvl_abc"
type="abc_nvl_streetnamepercentage" indexed="true" stored="false"/>
<dynamicField name="*streetnamepercentage.typo_abc"
type="abc_streetnamepercentage.typo" indexed="true" stored="false"/>
<dynamicField name="*streetnamepercentage.typo_nvl_abc"
type="abc_nvl_streetnamepercentage.typo" indexed="true" stored="false"/>
<dynamicField name="*streetnamematchnullone_abc"
type="abc_streetnamematchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetnamematchnullone_nvl_abc"
type="abc_nvl_streetnamematchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetnamematchnullboth_abc"
type="abc_streetnamematchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetnamematchnullboth_nvl_abc"
type="abc_nvl_streetnamematchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetnamesthes.soundex_abc"
type="abc_streetnamesthes_soundex" indexed="true" stored="false"/>
<dynamicField name="*streetnamesthes_abc" type="abc_streetnamesthes"
indexed="true" stored="false"/>
<dynamicField name="*streetnamesthes_nvl_abc"
type="abc_nvl_streetnamesthes" indexed="true" stored="false"/>
<!-- streetnumber / Street Number -->
<dynamicField name="*streetnumberpercentage_abc"
type="abc_streetnumberpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetnumberpercentage_nvl_abc"
type="abc_nvl_streetnumberpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetnumbermatchnullone_abc"
type="abc_streetnumbermatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetnumbermatchnullone_nvl_abc"
type="abc_nvl_streetnumbermatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetnumbermatchnullboth_abc"
type="abc_streetnumbermatchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetnumbermatchnullboth_nvl_abc"
type="abc_nvl_streetnumbermatchnullboth" indexed="true" stored="false"/>
<!-- streetsecondary / Street Secondary -->
<dynamicField name="*streetsecondarypercentage_abc"
type="abc_streetsecondarypercentage" indexed="true" stored="false"/>
<dynamicField name="*streetsecondarypercentage_nvl_abc"
type="abc_nvl_streetsecondarypercentage" indexed="true" stored="false"/>
<dynamicField name="*streetsecondarymatchnullone_abc"
type="abc_streetsecondarymatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetsecondarymatchnullone_nvl_abc"
type="abc_nvl_streetsecondarymatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetsecondarymatchnullboth_abc"
type="abc_streetsecondarymatchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetsecondarymatchnullboth_nvl_abc"
type="abc_nvl_streetsecondarymatchnullboth" indexed="true" stored="false"/>
<!-- streetpredirectional / Street Predirectional / Street Pre Directional
-->
<dynamicField name="*streetpredirectionalpercentage_abc"
type="abc_streetpredirectionalpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetpredirectionalpercentage_nvl_abc"
type="abc_nvl_streetpredirectionalpercentage" indexed="true"
stored="false"/>
<dynamicField name="*streetpredirectionalmatchnullone_abc"
type="abc_streetpredirectionalmatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetpredirectionalmatchnullone_nvl_abc"
type="abc_nvl_streetpredirectionalmatchnullone" indexed="true"
stored="false"/>
<dynamicField name="*streetpredirectionalmatchnullboth_abc"
type="abc_streetpredirectionalmatchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetpredirectionalmatchnullboth_nvl_abc"
type="abc_nvl_streetpredirectionalmatchnullboth" indexed="true"
stored="false"/>
<!-- streetpostdirectional / Street Postdirectional / Street Post
Directional -->
<dynamicField name="*streetpostdirectionalpercentage_abc"
type="abc_streetpostdirectionalpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetpostdirectionalpercentage_nvl_abc"
type="abc_nvl_streetpostdirectionalpercentage" indexed="true"
stored="false"/>
<dynamicField name="*streetpostdirectionalmatchnullone_abc"
type="abc_streetpostdirectionalmatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetpostdirectionalmatchnullone_nvl_abc"
type="abc_nvl_streetpostdirectionalmatchnullone" indexed="true"
stored="false"/>
<dynamicField name="*streetpostdirectionalmatchnullboth_abc"
type="abc_streetpostdirectionalmatchnullboth" indexed="true"
stored="false"/>
<dynamicField name="*streetpostdirectionalmatchnullboth_nvl_abc"
type="abc_nvl_streetpostdirectionalmatchnullboth" indexed="true"
stored="false"/>
<!-- streetsuffix / Street Suffix -->
<dynamicField name="*streetsuffixpercentage_abc"
type="abc_streetsuffixpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetsuffixpercentage_nvl_abc"
type="abc_nvl_streetsuffixpercentage" indexed="true" stored="false"/>
<dynamicField name="*streetsuffixmatchnullone_abc"
type="abc_streetsuffixmatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetsuffixmatchnullone_nvl_abc"
type="abc_nvl_streetsuffixmatchnullone" indexed="true" stored="false"/>
<dynamicField name="*streetsuffixmatchnullboth_abc"
type="abc_streetsuffixmatchnullboth" indexed="true" stored="false"/>
<dynamicField name="*streetsuffixmatchnullboth_nvl_abc"
type="abc_nvl_streetsuffixmatchnullboth" indexed="true" stored="false"/>
<!-- pobox / PO Box -->
<dynamicField name="*poboxpercentage_abc" type="abc_poboxpercentage"
indexed="true" stored="false"/>
<dynamicField name="*poboxpercentage_nvl_abc"
type="abc_nvl_poboxpercentage" indexed="true" stored="false"/>
<dynamicField name="*poboxmatchnullone_abc" type="abc_poboxmatchnullone"
indexed="true" stored="false"/>
<dynamicField name="*poboxmatchnullone_nvl_abc"
type="abc_nvl_poboxmatchnullone" indexed="true" stored="false"/>
<dynamicField name="*poboxmatchnullboth_abc" type="abc_poboxmatchnullboth"
indexed="true" stored="false"/>
<dynamicField name="*poboxmatchnullboth_nvl_abc"
type="abc_nvl_poboxmatchnullboth" indexed="true" stored="false"/>
<!-- matchnullone matchnullboth / Match Null One / Match Null Both -->
<!-- TODO: not sure we still need these? Also in fieldType section. -->
<dynamicField name="*matchnullone_abc" type="abc_matchnullone"
indexed="true" stored="false"/>
<dynamicField name="*matchnullone_nvl_abc" type="abc_nvl_matchnullone"
indexed="true" stored="false"/>
<dynamicField name="*matchnullboth_abc" type="abc_matchnullboth"
indexed="true" stored="false"/>
<dynamicField name="*matchnullboth_nvl_abc" type="abc_nvl_matchnullboth"
indexed="true" stored="false"/>
<!-- Fields we're sure are text -->
<dynamicField name="*fulltext_abc" type="abc_fulltext" indexed="true"
stored="true"/>
<dynamicField name="*id_abc" type="abc_id" indexed="true" stored="true"/>
<!-- This was the early v4 Solr behavior, but now whole-field is default
-->
<!-- with stop words -->
<dynamicField name="*percentmatchanywordcorewordsonly.soundex_abc"
type="abc_word_based_stopwords_soundex" indexed="true" stored="true"/>
<dynamicField name="*percentmatchanywordcorewordsonly_nvl_abc"
type="abc_word_based_stopwords_nvl" indexed="true" stored="true"/>
<dynamicField name="*percentmatchanywordcorewordsonly_abc"
type="abc_word_based_stopwords" indexed="true" stored="true"/>
<!-- without stop words -->
<dynamicField name="*percentmatchanyword.soundex_abc"
type="abc_word_based_default_soundex" indexed="true" stored="true"/>
<dynamicField name="*percentmatchanyword_nvl_abc"
type="abc_word_based_nvl" indexed="true" stored="true"/>
<dynamicField name="*percentmatchanyword_abc"
type="abc_word_based_default" indexed="true" stored="true"/>
<!-- Main type always stored -->
<!-- IMPORTANT: Least specific patterns MUST BE LAST IN LIST!!! -->
<dynamicField name="*_nvl_abc" type="abc_nvl" indexed="true"
stored="true"/>
<!-- IMPORTANT: This is the default type "percentmatchwholefield" -->
<dynamicField name="*_abc" type="abc_default" indexed="true"
stored="true"/>
<!-- TODO: Eventually declare Variants NOT Stored, but stored is nice
for debugging -->
<dynamicField name="*_i" type="int" indexed="true" stored="true"/>
<dynamicField name="*_is" type="int" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_s" type="string" indexed="true" stored="true" />
<dynamicField name="*_ss" type="string" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_l" type="long" indexed="true" stored="true"/>
<dynamicField name="*_ls" type="long" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_t" type="text_general" indexed="true"
stored="true"/>
<dynamicField name="*_txt" type="text_general" indexed="true"
stored="true" multiValued="true"/>
<dynamicField name="*_en" type="text_en" indexed="true"
stored="true" multiValued="true"/>
<dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
<dynamicField name="*_bs" type="boolean" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_f" type="float" indexed="true" stored="true"/>
<dynamicField name="*_fs" type="float" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_d" type="double" indexed="true" stored="true"/>
<dynamicField name="*_ds" type="double" indexed="true" stored="true"
multiValued="true"/>
<!-- Type used to index the lat and lon components for the "location"
FieldType -->
<dynamicField name="*_coordinate" type="tdouble" indexed="true"
stored="false" />
<dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
<dynamicField name="*_dts" type="date" indexed="true" stored="true"
multiValued="true"/>
<dynamicField name="*_p" type="location" indexed="true" stored="true"/>
<!-- some trie-coded dynamic fields for faster range queries -->
<dynamicField name="*_ti" type="tint" indexed="true" stored="true"/>
<dynamicField name="*_tl" type="tlong" indexed="true" stored="true"/>
<dynamicField name="*_tf" type="tfloat" indexed="true" stored="true"/>
<dynamicField name="*_td" type="tdouble" indexed="true" stored="true"/>
<dynamicField name="*_tdt" type="tdate" indexed="true" stored="true"/>
<dynamicField name="*_pi" type="pint" indexed="true" stored="true"/>
<dynamicField name="*_c" type="currency" indexed="true"
stored="true"/>
<dynamicField name="ignored_*" type="ignored" multiValued="true"/>
<dynamicField name="attr_*" type="text_general" indexed="true"
stored="true" multiValued="true"/>
<dynamicField name="random_*" type="random" />
<!-- uncomment the following to ignore any fields that don't already
match an existing
field name or dynamic field, rather than reporting them as an
error.
alternately, change the type="ignored" to some other type e.g.
"text" if you want
unknown fields indexed and/or stored by default -->
<!--dynamicField name="*" type="ignored" multiValued="true" /-->
</fields>
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a
required field
-->
<uniqueKey>id</uniqueKey>
<!-- DEPRECATED: The defaultSearchField is consulted by various query
parsers when
parsing a query string that isn't explicit about the field. Machine
(non-user)
generated queries are best made explicit, or they can use the "df"
request parameter
which takes precedence over this.
Note: Un-commenting defaultSearchField will be insufficient if your
request handler
in solrconfig.xml defines "df", which takes precedence. That would need
to be removed.
<defaultSearchField>text</defaultSearchField> -->
<!-- DEPRECATED: The defaultOperator (AND|OR) is consulted by various
query parsers
when parsing a query string to determine if a clause of the query should
be marked as
required or optional, assuming the clause isn't already marked by some
operator.
The default is OR, which is generally assumed so it is not a good idea to
change it
globally here. The "q.op" request parameter takes precedence over this.
<solrQueryParser defaultOperator="OR"/> -->
<!-- copyField commands copy one field to another at the time a document
is added to the index. It's used either to index the same field
differently,
or to add multiple fields to the same field for easier/faster
searching. -->
<copyField source="cat" dest="text"/>
<copyField source="name" dest="text"/>
<copyField source="manu" dest="text"/>
<copyField source="features" dest="text"/>
<copyField source="includes" dest="text"/>
<copyField source="manu" dest="manu_exact"/>
<!-- Copy the price into a currency enabled field (default USD) -->
<copyField source="price" dest="price_c"/>
<!-- Text fields from SolrCell to search by default in our catch-all
field -->
<copyField source="title" dest="text"/>
<copyField source="author" dest="text"/>
<copyField source="description" dest="text"/>
<copyField source="keywords" dest="text"/>
<copyField source="content" dest="text"/>
<copyField source="content_type" dest="text"/>
<copyField source="resourcename" dest="text"/>
<copyField source="url" dest="text"/>
<!-- Create a string version of author for faceting -->
<copyField source="author" dest="author_s"/>
<!-- Above, multiple source fields are copied to the [text] field.
Another way to map multiple source fields to the same
destination field is to use the dynamic field syntax.
copyField also supports a maxChars to copy setting. -->
<!-- <copyField source="*_t" dest="text" maxChars="3000"/> -->
<!-- copy name to alphaNameSort, a field designed for sorting by name -->
<!-- <copyField source="name" dest="alphaNameSort"/> -->
<!-- Abc Copy Fields -->
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.case_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.case.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.case_nvl_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.string_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.string.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.string_nvl_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.RECORD_ID_abc"
dest="ADDRESS_ALL.RECORD_ID_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.typo_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.string_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.string.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.string_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_TYPE_CD_abc"
dest="ADDRESS_ALL.ADDR_TYPE_CD_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorecase_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.ignorepunctuation_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.firstnamesthes_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.firstnamesthes.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.firstnamesthes_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.typo_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.string_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.string.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.string_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.LONGITUDE_abc"
dest="ADDRESS_ALL.LONGITUDE_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorecase_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorepunctuation_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorepunctuation.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.ignorepunctuation_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.typo_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.string_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.string.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.string_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.LATITUDE_abc"
dest="ADDRESS_ALL.LATITUDE_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.ignorecase_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamesthes_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnumberpercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpredirectionalpercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetnamepercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsuffixpercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetpostdirectionalpercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.streetsecondarypercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.poboxpercentage_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.typo_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.string_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.string.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.string_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.ADDR_NAME_abc"
dest="ADDRESS_ALL.ADDR_NAME_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.ignorecase_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.citynamesthes_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.citynamesthes.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.citynamesthes_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc" dest="ADDRESS_ALL.CITY.typo_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.porterstem.ignorenonalphanumeric.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc" dest="ADDRESS_ALL.CITY.string_abc"
/>
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.string.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.string_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.CITY_abc"
dest="ADDRESS_ALL.CITY.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.CITY_abc" dest="ADDRESS_ALL.CITY_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.ignorecase_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.porterstem.ignorenonalphanumeric.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.string_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.string.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.string_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly.soundex_abc" />
<copyField source="ADDRESS_ALL.STATE_abc"
dest="ADDRESS_ALL.STATE.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.STATE_abc" dest="ADDRESS_ALL.STATE_nvl_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefield_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.ignorecase_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.soundex_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.typo_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.typo.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.typo_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric_nvl_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchwholefieldcorewordsonly.porterstem.ignorenonalphanumeric.soundex_nvl_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.string_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.string.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.string_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword.soundex_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanyword_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly.soundex_abc"
/>
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR.percentmatchanywordcorewordsonly_nvl_abc" />
<copyField source="ADDRESS_ALL.EMAIL_ADDR_abc"
dest="ADDRESS_ALL.EMAIL_ADDR_nvl_abc" />
<types>
<!-- field type definitions. The "name" attribute is
just a label to be used by field definitions. The "class"
attribute and any other attributes determine the real
behavior of the fieldType.
Class names starting with "solr" refer to java classes in a
standard package such as org.apache.solr.analysis
-->
<!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
<fieldType name="string" class="solr.StrField" sortMissingLast="true" />
<!-- boolean type: "true" or "false" -->
<fieldType name="boolean" class="solr.BoolField"
sortMissingLast="true"/>
<!-- sortMissingLast and sortMissingFirst attributes are optional
attributes are
currently supported on types that are sorted internally as strings
and on numeric types.
This includes "string","boolean", and, as of 3.5 (and 4.x),
int, float, long, date, double, including the "Trie" variants.
- If sortMissingLast="true", then a sort on this field will cause
documents
without the field to come after documents with the field,
regardless of the requested sort order (asc or desc).
- If sortMissingFirst="true", then a sort on this field will cause
documents
without the field to come before documents with the field,
regardless of the requested sort order.
- If sortMissingLast="false" and sortMissingFirst="false" (the
default),
then default lucene sorting will be used which places docs without
the
field first in an ascending sort and last in a descending sort.
-->
<!--
Default numeric field types. For faster range queries, consider the
tint/tfloat/tlong/tdouble types.
-->
<fieldType name="int" class="solr.TrieIntField" precisionStep="0"
positionIncrementGap="0"/>
<fieldType name="float" class="solr.TrieFloatField" precisionStep="0"
positionIncrementGap="0"/>
<fieldType name="long" class="solr.TrieLongField" precisionStep="0"
positionIncrementGap="0"/>
<fieldType name="double" class="solr.TrieDoubleField" precisionStep="0"
positionIncrementGap="0"/>
<!--
Numeric field types that index each value at various levels of
precision
to accelerate range queries when the number of values between the range
endpoints is large. See the javadoc for NumericRangeQuery for internal
implementation details.
Smaller precisionStep values (specified in bits) will lead to more
tokens
indexed per value, slightly larger index size, and faster range
queries.
A precisionStep of 0 disables indexing at different precision levels.
-->
<fieldType name="tint" class="solr.TrieIntField" precisionStep="8"
positionIncrementGap="0"/>
<fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8"
positionIncrementGap="0"/>
<fieldType name="tlong" class="solr.TrieLongField" precisionStep="8"
positionIncrementGap="0"/>
<fieldType name="tdouble" class="solr.TrieDoubleField"
precisionStep="8" positionIncrementGap="0"/>
<!-- The format for this date field is of the form
1995-12-31T23:59:59Z, and
is a more restricted form of the canonical representation of
dateTime
http://www.w3.org/TR/xmlschema-2/#dateTime
The trailing "Z" designates UTC time and is mandatory.
Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
All other components are mandatory.
Expressions can also be used to denote calculations that should be
performed relative to "NOW" to determine the value, ie...
NOW/HOUR
... Round to the start of the current hour
NOW-1DAY
... Exactly 1 day prior to now
NOW/DAY+6MONTHS+3DAYS
... 6 months and 3 days in the future from the start of
the current day
Consult the DateField javadocs for more information.
Note: For faster range queries, consider the tdate type
-->
<fieldType name="date" class="solr.TrieDateField" precisionStep="0"
positionIncrementGap="0"/>
<!-- A Trie based date field for faster date range queries and date
faceting. -->
<fieldType name="tdate" class="solr.TrieDateField" precisionStep="6"
positionIncrementGap="0"/>
<!--Binary data type. The data should be sent/retrieved in as Base64
encoded Strings -->
<fieldType name="binary" class="solr.BinaryField"/>
<!--
Note:
These should only be used for compatibility with existing indexes
(created with lucene or older Solr versions).
Use Trie based fields instead. As of Solr 3.5 and 4.x, Trie based
fields support sortMissingFirst/Last
Plain numeric field types that store and index the text
value verbatim (and hence don't correctly support range queries,
since the
lexicographic ordering isn't equal to the numeric ordering)
-->
<fieldType name="pint" class="solr.IntField"/>
<fieldType name="plong" class="solr.LongField"/>
<fieldType name="pfloat" class="solr.FloatField"/>
<fieldType name="pdouble" class="solr.DoubleField"/>
<fieldType name="pdate" class="solr.DateField" sortMissingLast="true"/>
<!-- The "RandomSortField" is not used to store or search any
data. You can declare fields of this type it in your schema
to generate pseudo-random orderings of your docs for sorting
or function purposes. The ordering is generated based on the field
name and the version of the index. As long as the index version
remains unchanged, and the same field name is reused,
the ordering of the docs will be consistent.
If you want different psuedo-random orderings of documents,
for the same version of the index, use a dynamicField and
change the field name in the request.
-->
<fieldType name="random" class="solr.RandomSortField" indexed="true" />
<!-- Abc Types -->
<!-- ========== -->
<!-- Most general fields will be IDs
So basing this more closely on "string"
-->
<!--
<fieldType name="abc_id" class="solr.StrField" sortMissingLast="true"
positionIncrementGap="100" />
-->
<fieldType name="abc_id" class="solr.TextField" sortMissingLast="true">
<analyzer>
<!-- Keyword doesn't actually tokenize anything -->
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_fulltext" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Glue everything left into one giant token -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- If you change this, see also FirstFiveDigitsUtil and names for USA
-->
<fieldType name="abc_default" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Glue everything left into one giant token -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Generic Null Value / Non Null Value -->
<!-- Some types, such as Addresses, have their own nvl types -->
<fieldType name="abc_nvl" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_word_based_default" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_word_based_default_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Generic Null Value / Non Null Value -->
<!-- Some types, such as Addresses, have their own nvl types -->
<fieldType name="abc_word_based_nvl" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_word_based_stopwords" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
</analyzer>
</fieldType>
<fieldType name="abc_word_based_stopwords_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Generic Null Value / Non Null Value -->
<!-- Some types, such as Addresses, have their own nvl types -->
<fieldType name="abc_word_based_stopwords_nvl" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_ignorecase" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_ignorepunctuation" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory" />
<!--<filter class="solr.WordDelimiterFilterFactory" />-->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_ignorepunctuation_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory" />
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!--<filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
inject="false" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_porterstem" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!-- <tokenizer class="solr.WhitespaceTokenizerFactory" /> -->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Need Lowercase before Snowball -->
<filter class="solr.LowerCaseFilterFactory"/>
<!--<filter class="solr.PorterStemFilterFactory" />-->
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_porterstem_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_firstfivedigits" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!-- Keyword doesn't actually tokenize anything -->
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- discard anything that's not 5 digits
including nuke the entire field if not starting with 5 digits
-->
<filter class="solr.PatternReplaceFilterFactory" pattern="^([0-9]{5})?.*"
replacement="$1" replace="all" />
<!--<filter class="com.abc.fulltext.multiop.FirstFiveDigitsFilterFactory"
/>-->
<!-- Step 1: Remove everything that's not a digit -->
<!--<filter class="solr.PatternReplaceFilterFactory" pattern="[^0-9]"
replacement="" replace="all" />-->
<!-- Step 2: Keep only first 5 digits -->
<!--<filter class="solr.PatternReplaceFilterFactory"
pattern="^([0-9]{1,5}).*" replacement="$1" replace="all" />-->
<!-- Step 3: Discard anything not 5 digits -->
<!--<filter class="solr.LengthFilterFactory" min="5" max="5" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- No lowercase since just digits -->
</analyzer>
</fieldType>
<fieldType name="abc_ignorenonalphanumeric" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!-- Keyword doesn't actually tokenize anything -->
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!--<filter
class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
<!-- throw out all everything except a-z and 0-9, include A-Z just to be
safe -->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Don't need to glue back together since used Keyword -->
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_ignorenonalphanumeric_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_typo" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<!-- <tokenizer class="solr.WhitespaceTokenizerFactory" /> -->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!--<filter class="solr.PhoneticFilterFactory" encoder="Metaphone" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_porterstem_ignorenonalphanumeric"
class="solr.TextField" positionIncrementGap="100">
<analyzer>
<!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<!-- Need Lowercase before Snowball -->
<filter class="solr.LowerCaseFilterFactory"/>
<!--<filter class="solr.PorterStemFilterFactory" />-->
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<!--<filter
class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
<!-- Throw out all everything except a-z and 0-9, include A-Z just to be
safe -->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Glue all the tokens back together -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_porterstem_ignorenonalphanumeric_soundex"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- Need Lowercase before Snowball -->
<filter class="solr.LowerCaseFilterFactory"/>
<!--<filter class="solr.PorterStemFilterFactory" />-->
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<!--<filter
class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
<!-- throw out all everything except a-z and 0-9, include A-Z just to be
safe -->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Glue all the tokens back together -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!--<filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
inject="false" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_stopwords_porterstem_ignorenonalphanumeric"
class="solr.TextField" positionIncrementGap="100">
<analyzer>
<!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
<!-- Need Lowercase before Snowball -->
<filter class="solr.LowerCaseFilterFactory"/>
<!--<filter class="solr.PorterStemFilterFactory" />-->
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<!--<filter
class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
<!-- throw out all everything except a-z and 0-9, include A-Z just to be
safe -->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Glue all the tokens back together -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_stopwords_porterstem_ignorenonalphanumeric_soundex"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<!--<tokenizer class="solr.KeywordTokenizerFactory"/>-->
<!--<tokenizer class="solr.WhitespaceTokenizerFactory" />-->
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
<!-- Need Lowercase before Snowball -->
<filter class="solr.LowerCaseFilterFactory"/>
<!--<filter class="solr.PorterStemFilterFactory" />-->
<filter class="solr.SnowballPorterFilterFactory" language="English" />
<!--<filter
class="com.abc.fulltext.multiop.IgnoreNonAlphaNumericFilterFactory" />-->
<!-- Throw out all everything except a-z and 0-9, include A-Z just to be
safe -->
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-zA-Z0-9])"
replacement="" replace="all" />
<!-- Glue all the tokens back together -->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- <filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex"
inject="false" /> -->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_stopwords" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_stopwords_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/stopwords.txt" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_junk" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/junk.txt" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_profanity" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="wordlists/profanity.txt" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_areacodethes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[^0-9]" />
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/areacodethes.txt" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_citynamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/citynamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/citynamesthes_abbrev.txt" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_citynamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/citynamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/citynamesthes_abbrev.txt" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_companynamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/companynamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/companynamesthes_abbrev.txt" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_companynamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/companynamesthes_expand.txt" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_countrynamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/countrynamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/countrynamesthes_abbrev.txt" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_countrynamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/countrynamesthes_expand.txt" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_firstnamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/firstnamesthes.txt" />
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_firstnamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/firstnamesthes.txt" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_provincenamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/provincenamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/provincenamesthes_abbrev.txt" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_provincenamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/provincenamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/provincenamesthes_abbrev.txt" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- thes changed to index and query time due to concatenation -->
<fieldType name="abc_statenamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="query">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/statenamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/statenamesthes_abbrev.txt" />-->
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_statenamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.PatternTokenizerFactory"
pattern="[-
\t\r\n,!?:;`"*=/\\()\[\]\{\}<>#$%\^&~]"
/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/statenamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/statenamesthes_abbrev.txt" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- some other thresaurus items in the address section -->
<!--
*********************************************
Begin Addresses: Abc fieldTypes
*********************************************
streetname
streetnumber
streetsecondary
streetpredirectional
streetpostdirectional
streetsuffix
pobox
matchnullone matchnullboth
-->
<!-- Not using
<filter class="com.abc.fulltext.multiop.ConcatFilterFactory" />
Because we're using
<tokenizer class="solr.KeywordTokenizerFactory"/>
which means already only one token
unless stop words or thes or whatever is used
-->
<!-- streetname / Street Name -->
<fieldType name="abc_streetnamepercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamepercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnamepercentage_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamepercentage" />
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<!-- Keep LowerCase last due to Soundex -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnamepercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamepercentage" />
<!-- Only added at Index Time -->
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnamepercentage.typo" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamepercentage.typo" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnamepercentage.typo" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamepercentage.typo" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnamematchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamematchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnamematchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamematchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnamematchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamematchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnamematchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamematchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- TODO: plural vs. singular? abc_streetnamesthes vs. abc_streetnamethes
-->
<fieldType name="abc_streetnamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamesthes" />
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/streetnamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
<!-- TODO: consider concat filter -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/streetnamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
<!-- TODO: consider concat filter -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnamesthes_soundex" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamesthes" />
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/streetnamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
<!-- Replace numbers/digits with words -->
<filter class="solr.PatternReplaceFilterFactory" pattern="0"
replacement="zero" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="1"
replacement="one" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="2"
replacement="two" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="3"
replacement="three" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="4"
replacement="four" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="5"
replacement="five" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="6"
replacement="six" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="7"
replacement="seven" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="8"
replacement="eight" replace="all" />
<filter class="solr.PatternReplaceFilterFactory" pattern="9"
replacement="nine" replace="all" />
<filter class="solr.PhoneticFilterFactory"
encoder="DoubleMetaphone"
inject="false"
maxCodeLength="25"
/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnamesthes" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnamesthes" />
<!-- TODO: Break up multi word street names, etc. at this point, need token
filter, not tokenizer -->
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- TODO: Break up multi word street names, etc. at this point, need token
filter, not tokenizer -->
<filter class="solr.SynonymFilterFactory" ignoreCase="true" expand="false"
synonyms="wordlists/streetnamesthes_expand.txt" />
<!--<filter class="solr.SynonymFilterFactory" ignoreCase="true"
expand="false" synonyms="wordlists/streetnamesthes_abbrev.txt" />-->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- streetnumber / Street Number -->
<fieldType name="abc_streetnumberpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumberpercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnumberpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumberpercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnumbermatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumbermatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnumbermatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumbermatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetnumbermatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumbermatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetnumbermatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetnumbermatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- streetsecondary / Street Secondary -->
<fieldType name="abc_streetsecondarypercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarypercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsecondarypercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarypercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetsecondarymatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarymatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsecondarymatchnullone"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarymatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetsecondarymatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarymatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsecondarymatchnullboth"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsecondarymatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- streetpredirectional / Street Predirectional / Street Pre Directional
-->
<fieldType name="abc_streetpredirectionalpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalpercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpredirectionalpercentage"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalpercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetpredirectionalmatchnullone"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalmatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpredirectionalmatchnullone"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalmatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetpredirectionalmatchnullboth"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalmatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpredirectionalmatchnullboth"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpredirectionalmatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- streetpostdirectional / Street Postdirectional / Street Post
Directional -->
<fieldType name="abc_streetpostdirectionalpercentage"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalpercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpostdirectionalpercentage"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalpercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetpostdirectionalmatchnullone"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalmatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpostdirectionalmatchnullone"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalmatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetpostdirectionalmatchnullboth"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalmatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetpostdirectionalmatchnullboth"
class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetpostdirectionalmatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- streetsuffix / Street Suffix -->
<fieldType name="abc_streetsuffixpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixpercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsuffixpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixpercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetsuffixmatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixmatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsuffixmatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixmatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_streetsuffixmatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixmatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_streetsuffixmatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="streetsuffixmatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- pobox / PO Box -->
<fieldType name="abc_poboxpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxpercentage" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_poboxpercentage" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxpercentage" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_poboxmatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxmatchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_poboxmatchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxmatchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_poboxmatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxmatchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_poboxmatchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="poboxmatchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- matchnullone matchnullboth / Match Null One / Match Null Both -->
<!-- TODO: not sure we still need these? Also in dynamicField section -->
<fieldType name="abc_matchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="matchnullone" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_matchnullone" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="matchnullone" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- matchnullboth Match Null Both -->
<fieldType name="abc_matchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="matchnullboth" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="abc_nvl_matchnullboth" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="com.abc.fulltext.multiop.MDFilterSchemaFactory"
operator="matchnullboth" />
<filter class="com.abc.fulltext.multiop.NullValueFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- solr.TextField allows the specification of custom text analyzers
specified as a tokenizer and a list of token filters. Different
analyzers may be specified for indexing and querying.
The optional positionIncrementGap puts space between multiple
fields of
this type on the same document, with the purpose of preventing
false phrase
matching across fields.
For more info on customizing your analyzer chain, please see
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
-->
<!-- One can also specify an existing Analyzer class that has a
default constructor via the class attribute on the analyzer
element.
Example:
<fieldType name="text_greek" class="solr.TextField">
<analyzer class="org.apache.lucene.analysis.el.GreekAnalyzer"/>
</fieldType>
-->
<!-- A text field that only splits on whitespace for exact matching of
words -->
<fieldType name="text_ws" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
</analyzer>
</fieldType>
<!-- A general text field that has reasonable, generic
cross-language defaults: it tokenizes with StandardTokenizer,
removes stop words from case-insensitive "stopwords.txt"
(empty by default), and down cases. At query time only, it
also applies synonyms. -->
<fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- A text field with defaults appropriate for English: it
tokenizes with StandardTokenizer, removes English stop words
(lang/stopwords_en.txt), down cases, protects words from
protwords.txt, and
finally applies Porter's stemming. The query time analyzer
also applies synonyms from synonyms.txt. -->
<fieldType name="text_en" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<!-- Case insensitive stop word removal.
add enablePositionIncrements=true in both the index and query
analyzers to leave a 'gap' for more accurate phrase queries.
-->
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="lang/stopwords_en.txt"
enablePositionIncrements="true"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<!-- Optionally you may want to use this less aggressive stemmer instead of
PorterStemFilterFactory:
<filter class="solr.EnglishMinimalStemFilterFactory"/>
-->
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="lang/stopwords_en.txt"
enablePositionIncrements="true"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<!-- Optionally you may want to use this less aggressive stemmer instead of
PorterStemFilterFactory:
<filter class="solr.EnglishMinimalStemFilterFactory"/>
-->
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- A text field with defaults appropriate for English, plus
aggressive word-splitting and autophrase features enabled.
This field is just like text_en, except it adds
WordDelimiterFilter to enable splitting and matching of
words on case-change, alpha numeric boundaries, and
non-alphanumeric chars. This means certain compound word
cases will work, for example query "wi fi" will match
document "WiFi" or "wi-fi".
-->
<fieldType name="text_en_splitting" class="solr.TextField"
positionIncrementGap="100" autoGeneratePhraseQueries="true">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<!-- Case insensitive stop word removal.
add enablePositionIncrements=true in both the index and query
analyzers to leave a 'gap' for more accurate phrase queries.
-->
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="lang/stopwords_en.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="lang/stopwords_en.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Less flexible matching, but less false matches. Probably not
ideal for product names,
but may be good for SKUs. Can insert dashes in the wrong place
and still match. -->
<fieldType name="text_en_splitting_tight" class="solr.TextField"
positionIncrementGap="100" autoGeneratePhraseQueries="true">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="false"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_en.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="0" catenateWords="1"
catenateNumbers="1" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.EnglishMinimalStemFilterFactory"/>
<!-- this filter can remove any duplicate tokens that appear at the
same position - sometimes
possible with WordDelimiterFilter in conjuncton with stemming.
-->
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
<!-- Just like text_general except it reverses the characters of
each token, to enable more efficient leading wildcard queries. -->
<fieldType name="text_general_rev" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ReversedWildcardFilterFactory"
withOriginal="true"
maxPosAsterisk="3" maxPosQuestion="2"
maxFractionAsterisk="0.33"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- charFilter + WhitespaceTokenizer -->
<!--
<fieldType name="text_char_norm" class="solr.TextField"
positionIncrementGap="100" >
<analyzer>
<charFilter class="solr.MappingCharFilterFactory"
mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
</analyzer>
</fieldType>
-->
<!-- This is an example of using the KeywordTokenizer along
With various TokenFilterFactories to produce a sortable field
that does not include some properties of the source text
-->
<fieldType name="alphaOnlySort" class="solr.TextField"
sortMissingLast="true" omitNorms="true">
<analyzer>
<!-- KeywordTokenizer does no actual tokenizing, so the entire
input string is preserved as a single token
-->
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- The LowerCase TokenFilter does what you expect, which can be
when you want your sorting to be case insensitive
-->
<filter class="solr.LowerCaseFilterFactory" />
<!-- The TrimFilter removes any leading or trailing whitespace -->
<filter class="solr.TrimFilterFactory" />
<!-- The PatternReplaceFilter gives you the flexibility to use
Java Regular expression to replace any sequence of characters
matching a pattern with an arbitrary replacement string,
which may include back references to portions of the original
string matched by the pattern.
See the Java Regular Expression documentation for more
information on pattern and replacement string syntax.
http://java.sun.com/j2se/1.6.0/docs/api/java/util/regex/package-summary.html
-->
<filter class="solr.PatternReplaceFilterFactory"
pattern="([^a-z])" replacement="" replace="all"
/>
</analyzer>
</fieldType>
<fieldType name="phonetic" stored="false" indexed="true"
class="solr.TextField" >
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.DoubleMetaphoneFilterFactory" inject="false"/>
</analyzer>
</fieldType>
<fieldType name="payloads" stored="false" indexed="true"
class="solr.TextField" >
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!--
The DelimitedPayloadTokenFilter can put payloads on tokens... for
example,
a token of "foo|1.4" would be indexed as "foo" with a payload of
1.4f
Attributes of the DelimitedPayloadTokenFilterFactory :
"delimiter" - a one character delimiter. Default is | (pipe)
"encoder" - how to encode the following value into a playload
float -> org.apache.lucene.analysis.payloads.FloatEncoder,
integer -> o.a.l.a.p.IntegerEncoder
identity -> o.a.l.a.p.IdentityEncoder
Fully Qualified class name implementing PayloadEncoder, Encoder
must have a no arg constructor.
-->
<filter class="solr.DelimitedPayloadTokenFilterFactory"
encoder="float"/>
</analyzer>
</fieldType>
<!-- lowercases the entire field value, keeping it as a single token.
-->
<fieldType name="lowercase" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
</fieldType>
<!--
Example of using PathHierarchyTokenizerFactory at index time, so
queries for paths match documents at that path, or in descendent paths
-->
<fieldType name="descendent_path" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory" />
</analyzer>
</fieldType>
<!--
Example of using PathHierarchyTokenizerFactory at query time, so
queries for paths match documents at that path, or in ancestor paths
-->
<fieldType name="ancestor_path" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
</analyzer>
</fieldType>
<!-- since fields of this type are by default not stored or indexed,
any data added to them will be ignored outright. -->
<fieldType name="ignored" stored="false" indexed="false"
multiValued="true" class="solr.StrField" />
<!-- This point type indexes the coordinates as separate fields
(subFields)
If subFieldType is defined, it references a type, and a dynamic field
definition is created matching *___<typename>. Alternately, if
subFieldSuffix is defined, that is used to create the subFields.
Example: if subFieldType="double", then the coordinates would be
indexed in fields myloc_0___double,myloc_1___double.
Example: if subFieldSuffix="_d" then the coordinates would be indexed
in fields myloc_0_d,myloc_1_d
The subFields are an implementation detail of the fieldType, and end
users normally should not need to know about them.
-->
<fieldType name="point" class="solr.PointType" dimension="2"
subFieldSuffix="_d"/>
<!-- A specialized field for geospatial search. If indexed, this
fieldType must not be multivalued. -->
<fieldType name="location" class="solr.LatLonType"
subFieldSuffix="_coordinate"/>
<!--
A Geohash is a compact representation of a latitude longitude pair in a
single field.
See http://wiki.apache.org/solr/SpatialSearch
-->
<fieldType name="geohash" class="solr.GeoHashField"/>
<!-- Money/currency field type. See
http://wiki.apache.org/solr/MoneyFieldType
Parameters:
defaultCurrency: Specifies the default currency if none
specified. Defaults to "USD"
precisionStep: Specifies the precisionStep for the TrieLong
field used for the amount
providerClass: Lets you plug in other exchange provider backend:
solr.FileExchangeRateProvider is the default and
takes one parameter:
currencyConfig: name of an xml file holding
exhange rates
solr.OpenExchangeRatesOrgProvider uses rates
from openexchangerates.org:
ratesFileLocation: URL or path to rates JSON
file (default latest.json on the web)
refreshInterval: Number of minutes between
each rates fetch (default: 1440, min: 60)
-->
<fieldType name="currency" class="solr.CurrencyField" precisionStep="8"
defaultCurrency="USD" currencyConfig="currency.xml" />
<!-- some examples for different languages (generally ordered by ISO
code) -->
<!-- Arabic -->
<fieldType name="text_ar" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- for any non-arabic -->
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ar.txt" enablePositionIncrements="true"/>
<!-- normalizes ﻯ to ﻱ, etc -->
<filter class="solr.ArabicNormalizationFilterFactory"/>
<filter class="solr.ArabicStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Bulgarian -->
<fieldType name="text_bg" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_bg.txt" enablePositionIncrements="true"/>
<filter class="solr.BulgarianStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Catalan -->
<fieldType name="text_ca" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- removes l', etc -->
<filter class="solr.ElisionFilterFactory" ignoreCase="true"
articles="lang/contractions_ca.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ca.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Catalan"/>
</analyzer>
</fieldType>
<!-- CJK bigram (see text_ja for a Japanese configuration using
morphological analysis) -->
<fieldType name="text_cjk" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- normalize width before bigram, as e.g. half-width dakuten
combine -->
<filter class="solr.CJKWidthFilterFactory"/>
<!-- for any non-CJK -->
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CJKBigramFilterFactory"/>
</analyzer>
</fieldType>
<!-- Czech -->
<fieldType name="text_cz" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_cz.txt" enablePositionIncrements="true"/>
<filter class="solr.CzechStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Danish -->
<fieldType name="text_da" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_da.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Danish"/>
</analyzer>
</fieldType>
<!-- German -->
<fieldType name="text_de" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_de.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.GermanNormalizationFilterFactory"/>
<filter class="solr.GermanLightStemFilterFactory"/>
<!-- less aggressive: <filter
class="solr.GermanMinimalStemFilterFactory"/> -->
<!-- more aggressive: <filter
class="solr.SnowballPorterFilterFactory" language="German2"/> -->
</analyzer>
</fieldType>
<!-- Greek -->
<fieldType name="text_el" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- greek specific lowercase for sigma -->
<filter class="solr.GreekLowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="false"
words="lang/stopwords_el.txt" enablePositionIncrements="true"/>
<filter class="solr.GreekStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Spanish -->
<fieldType name="text_es" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_es.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SpanishLightStemFilterFactory"/>
<!-- more aggressive: <filter
class="solr.SnowballPorterFilterFactory" language="Spanish"/> -->
</analyzer>
</fieldType>
<!-- Basque -->
<fieldType name="text_eu" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_eu.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory" language="Basque"/>
</analyzer>
</fieldType>
<!-- Persian -->
<fieldType name="text_fa" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<!-- for ZWNJ -->
<charFilter class="solr.PersianCharFilterFactory"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ArabicNormalizationFilterFactory"/>
<filter class="solr.PersianNormalizationFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_fa.txt" enablePositionIncrements="true"/>
</analyzer>
</fieldType>
<!-- Finnish -->
<fieldType name="text_fi" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_fi.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Finnish"/>
<!-- less aggressive: <filter
class="solr.FinnishLightStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- French -->
<fieldType name="text_fr" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- removes l', etc -->
<filter class="solr.ElisionFilterFactory" ignoreCase="true"
articles="lang/contractions_fr.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_fr.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.FrenchLightStemFilterFactory"/>
<!-- less aggressive: <filter
class="solr.FrenchMinimalStemFilterFactory"/> -->
<!-- more aggressive: <filter
class="solr.SnowballPorterFilterFactory" language="French"/> -->
</analyzer>
</fieldType>
<!-- Irish -->
<fieldType name="text_ga" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- removes d', etc -->
<filter class="solr.ElisionFilterFactory" ignoreCase="true"
articles="lang/contractions_ga.txt"/>
<!-- removes n-, etc. position increments is intentionally false!
-->
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/hyphenations_ga.txt" enablePositionIncrements="false"/>
<filter class="solr.IrishLowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ga.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory" language="Irish"/>
</analyzer>
</fieldType>
<!-- Galician -->
<fieldType name="text_gl" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_gl.txt" enablePositionIncrements="true"/>
<filter class="solr.GalicianStemFilterFactory"/>
<!-- less aggressive: <filter
class="solr.GalicianMinimalStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Hindi -->
<fieldType name="text_hi" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<!-- normalizes unicode representation -->
<filter class="solr.IndicNormalizationFilterFactory"/>
<!-- normalizes variation in spelling -->
<filter class="solr.HindiNormalizationFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_hi.txt" enablePositionIncrements="true"/>
<filter class="solr.HindiStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Hungarian -->
<fieldType name="text_hu" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_hu.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Hungarian"/>
<!-- less aggressive: <filter
class="solr.HungarianLightStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Armenian -->
<fieldType name="text_hy" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_hy.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Armenian"/>
</analyzer>
</fieldType>
<!-- Indonesian -->
<fieldType name="text_id" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_id.txt" enablePositionIncrements="true"/>
<!-- for a less aggressive approach (only inflectional suffixes),
set stemDerivational to false -->
<filter class="solr.IndonesianStemFilterFactory"
stemDerivational="true"/>
</analyzer>
</fieldType>
<!-- Italian -->
<fieldType name="text_it" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<!-- removes l', etc -->
<filter class="solr.ElisionFilterFactory" ignoreCase="true"
articles="lang/contractions_it.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_it.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.ItalianLightStemFilterFactory"/>
<!-- more aggressive: <filter
class="solr.SnowballPorterFilterFactory" language="Italian"/> -->
</analyzer>
</fieldType>
<!-- Japanese using morphological analysis (see text_cjk for a
configuration using bigramming)
NOTE: If you want to optimize search for precision, use default
operator AND in your query
parser config with <solrQueryParser defaultOperator="AND"/>
further down in this file. Use
OR if you would like to optimize for recall (default).
-->
<fieldType name="text_ja" class="solr.TextField"
positionIncrementGap="100" autoGeneratePhraseQueries="false">
<analyzer>
<!-- Kuromoji Japanese morphological analyzer/tokenizer
(JapaneseTokenizer)
Kuromoji has a search mode (default) that does segmentation
useful for search. A heuristic
is used to segment compounds into its parts and the compound
itself is kept as synonym.
Valid values for attribute mode are:
normal: regular segmentation
search: segmentation useful for search with synonyms
compounds (default)
extended: same as search mode, but unigrams unknown words
(experimental)
For some applications it might be good to use search mode for
indexing and normal mode for
queries to reduce recall and prevent parts of compounds from
being matched and highlighted.
Use <analyzer type="index"> and <analyzer type="query"> for this
and mode normal in query.
Kuromoji also has a convenient user dictionary feature that
allows overriding the statistical
model with your own entries for segmentation, part-of-speech
tags and readings without a need
to specify weights. Notice that user dictionaries have not been
subject to extensive testing.
User dictionary attributes are:
userDictionary: user dictionary filename
userDictionaryEncoding: user dictionary encoding (default is
UTF-8)
See lang/userdict_ja.txt for a sample user dictionary file.
Punctuation characters are discarded by default. Use
discardPunctuation="false" to keep them.
See http://wiki.apache.org/solr/JapaneseLanguageSupport for more
on Japanese language support.
-->
<tokenizer class="solr.JapaneseTokenizerFactory" mode="search"/>
<!--<tokenizer class="solr.JapaneseTokenizerFactory" mode="search"
userDictionary="lang/userdict_ja.txt"/>-->
<!-- Reduces inflected verbs and adjectives to their
base/dictionary forms (辞書形) -->
<filter class="solr.JapaneseBaseFormFilterFactory"/>
<!-- Removes tokens with certain part-of-speech tags -->
<filter class="solr.JapanesePartOfSpeechStopFilterFactory"
tags="lang/stoptags_ja.txt" enablePositionIncrements="true"/>
<!-- Normalizes full-width romaji to half-width and half-width kana
to full-width (Unicode NFKC subset) -->
<filter class="solr.CJKWidthFilterFactory"/>
<!-- Removes common tokens typically not useful for search, but
have a negative effect on ranking -->
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ja.txt" enablePositionIncrements="true" />
<!-- Normalizes common katakana spelling variations by removing any
last long sound character (U+30FC) -->
<filter class="solr.JapaneseKatakanaStemFilterFactory"
minimumLength="4"/>
<!-- Lower-cases romaji characters -->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Latvian -->
<fieldType name="text_lv" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_lv.txt" enablePositionIncrements="true"/>
<filter class="solr.LatvianStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- Dutch -->
<fieldType name="text_nl" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_nl.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.StemmerOverrideFilterFactory"
dictionary="lang/stemdict_nl.txt" ignoreCase="false"/>
<filter class="solr.SnowballPorterFilterFactory" language="Dutch"/>
</analyzer>
</fieldType>
<!-- Norwegian -->
<fieldType name="text_no" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_no.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Norwegian"/>
<!-- less aggressive: <filter
class="solr.NorwegianLightStemFilterFactory"/> -->
<!-- singular/plural: <filter
class="solr.NorwegianMinimalStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Portuguese -->
<fieldType name="text_pt" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_pt.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.PortugueseLightStemFilterFactory"/>
<!-- less aggressive: <filter
class="solr.PortugueseMinimalStemFilterFactory"/> -->
<!-- more aggressive: <filter
class="solr.SnowballPorterFilterFactory" language="Portuguese"/> -->
<!-- most aggressive: <filter
class="solr.PortugueseStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Romanian -->
<fieldType name="text_ro" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ro.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Romanian"/>
</analyzer>
</fieldType>
<!-- Russian -->
<fieldType name="text_ru" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_ru.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Russian"/>
<!-- less aggressive: <filter
class="solr.RussianLightStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Swedish -->
<fieldType name="text_sv" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_sv.txt" format="snowball"
enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Swedish"/>
<!-- less aggressive: <filter
class="solr.SwedishLightStemFilterFactory"/> -->
</analyzer>
</fieldType>
<!-- Thai -->
<fieldType name="text_th" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ThaiWordFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_th.txt" enablePositionIncrements="true"/>
</analyzer>
</fieldType>
<!-- Turkish -->
<fieldType name="text_tr" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.TurkishLowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="false"
words="lang/stopwords_tr.txt" enablePositionIncrements="true"/>
<filter class="solr.SnowballPorterFilterFactory"
language="Turkish"/>
</analyzer>
</fieldType>
</types>
<!-- Similarity is the scoring routine for each document vs. a query.
A custom Similarity or SimilarityFactory may be specified here, but
the default is fine for most applications.
For more info: http://wiki.apache.org/solr/SchemaXml#Similarity
-->
<!--
<similarity class="com.example.solr.CustomSimilarityFactory">
<str name="paramkey">param value</str>
</similarity>
-->
</schema>
On Fri, Apr 4, 2014 at 4:59 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> error.txt below
>
>
> Java Platform Detected x64
> Java Platform Detected -XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> -XX:+HeapDumpOnOutOfMemoryError -XX:+CreateMinidumpOnCrash
> 2014-04-04 15:49:43.341:INFO:oejs.Server:jetty-8.1.8.v20121106
> 2014-04-04 15:49:43.353:INFO:oejdp.ScanningAppProvider:Deployment monitor
> D:\AbcData\V12\application server\server\indexer\example\contexts at
> interval 0
> 2014-04-04 15:49:43.358:INFO:oejd.DeploymentManager:Deployable added:
> D:\AbcData\V12\application
> server\server\indexer\example\contexts\solr-jetty-context.xml
> 2014-04-04 15:49:43.989:INFO:oejw.StandardDescriptorProcessor:NO JSP
> Support for /solr, did not find org.apache.jasper.servlet.JspServlet
> Null identity service, trying login service: null
> Finding identity service: null
> 2014-04-04 15:49:44.011:INFO:oejsh.ContextHandler:started
> o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
> server\server\indexer\example/webapps/solr.war
> 2014-04-04 15:49:44.012:INFO:oejsh.ContextHandler:started
> o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
> server\server\indexer\example/webapps/solr.war
> Apr 04, 2014 3:49:44 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer$Initializer
> initialize
> INFO: looking for solr.xml: D:\AbcData\V12\application
> server\server\indexer\example\solr\solr.xml
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer <init>
> INFO: New CoreContainer 1879341237
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
> INFO: Loading CoreContainer using Solr Home: 'solr/'
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
> INFO: new SolrResourceLoader for directory: 'solr/'
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/apache-log4j-extras-1.2.17.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/jtds-1.2.5.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/log4j-1.2.17.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msbase.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/mssqlserver.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msutil.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/ojdbc6.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-api-1.7.5.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-nop-1.7.5.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/sqljdbc4.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting socketTimeout to: 0
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting urlScheme to: http://
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting connTimeout to: 0
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting maxConnectionsPerHost to: 20
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting corePoolSize to: 0
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting maximumPoolSize to: 2147483647
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting maxThreadIdleTime to: 5
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting sizeOfQueue to: -1
> Apr 04, 2014 3:49:44 PM
> org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
> INFO: Setting fairnessPolicy to: false
> Apr 04, 2014 3:49:44 PM org.apache.solr.client.solrj.impl.HttpClientUtil
> createClient
> INFO: Creating new http client,
> config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
> INFO: Registering Log Listener
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer create
> INFO: Creating SolrCore 'core0' using instanceDir: solr\core0
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
> INFO: new SolrResourceLoader for directory: 'solr\core0\'
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig initLibs
> INFO: Adding specified lib dirs to ClassLoader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-4.2.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-extras-4.2.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-core-0.7.2.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-dom-0.7.2.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcmail-jdk15-1.45.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcprov-jdk15-1.45.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/boilerpipe-1.1.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/commons-compress-1.4.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/dom4j-1.6.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/fontbox-1.7.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/icu4j-49.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/isoparser-1.0-RC-1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jdom-1.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jempbox-1.7.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/juniversalchardet-1.0.3.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/metadata-extractor-2.6.2.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/netcdf-4.2-min.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/pdfbox-1.7.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-3.8.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-3.8.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-schemas-3.8.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-scratchpad-3.8.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/rome-0.9.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tagsoup-1.2.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-core-1.3.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-parsers-1.3.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-core-0.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-tika-0.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xercesImpl-2.9.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xmlbeans-2.3.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xz-1.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/attributes-binder-1.0.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/carrot2-mini-3.6.2.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/hppc-0.4.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-core-asl-1.7.4.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-mapper-asl-1.7.4.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-collections-1.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-math-0.6.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/simple-xml-2.6.4.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/jsonic-1.2.7.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/langdetect-1.1-20120112.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-beanutils-1.7.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-collections-3.2.1.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-1.7.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-tools-2.0.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/common.jar' to
> classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/webapp.jar' to
> classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
> replaceClassLoader
> INFO: Adding
> 'file:/D:/AbcData/V12/applicationserver/server/default/lib/commons-pool-1.4.jar'
> to classloader
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
> INFO: Using Lucene MatchVersion: LUCENE_42
> Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
> INFO: Loaded SolrConfig: solrconfig.xml
> Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
> INFO: Reading Solr Schema
> Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
> INFO: Schema name=example
> Apr 04, 2014 3:49:46 PM org.apache.solr.schema.IndexSchema readSchema
> INFO: unique key field: id
> Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
> reload
> INFO: Reloading exchange rates from file currency.xml
> Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
> reload
> INFO: Reloading exchange rates from file currency.xml
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
> INFO: org.apache.solr.core.StandardDirectoryFactory
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
> INFO: [core0] Opening new SolrCore at solr\core0\,
> dataDir=D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
> INFO: JMX monitoring not detected for core: core0
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
> INFO: [core0] Added SolrEventListener for newSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[]}
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
> INFO: [core0] Added SolrEventListener for firstSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher
> warming in solrconfig.xml}]}
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
> INFO: org.apache.solr.core.StandardDirectoryFactory
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
> INFO: return new directory for D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data forceNew: false
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore getNewIndexDir
> INFO: New index directory detected: old=null
> new=D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index/
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initIndex
> WARNING: [core0] Solr index directory 'D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index' doesn't exist.
> Creating new index...
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
> INFO: return new directory for D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index forceNew: false
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy onCommit
> INFO: SolrDeletionPolicy.onCommit: commits:num=1
> commit{dir=D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1[segments_1]
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
> INFO: created json: solr.JSONResponseWriter
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
> INFO: adding lazy queryResponseWriter: solr.VelocityResponseWriter
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
> INFO: created velocity: solr.VelocityResponseWriter
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
> INFO: created xslt: solr.XSLTResponseWriter
> Apr 04, 2014 3:49:47 PM org.apache.solr.response.XSLTResponseWriter init
> INFO: xsltCacheLifetimeSeconds=5
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /dataimport:
> org.apache.solr.handler.dataimport.DataImportHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /select: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /query: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /browse: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /update: solr.XmlUpdateRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.extraction.ExtractingRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /update/extract: solr.extraction.ExtractingRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.FieldAnalysisRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /analysis/field: solr.FieldAnalysisRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /analysis/document: solr.DocumentAnalysisRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /admin/: solr.admin.AdminHandlers
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /admin/ping: solr.PingRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /debug/dump: solr.DumpRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.ReplicationHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /replication: solr.ReplicationHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /spell: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /tvrh: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /terms: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: adding lazy requestHandler: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
> initHandlersFromConfig
> INFO: created /elevate: solr.SearchHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.handler.loader.XMLLoader init
> INFO: xsltCacheLifetimeSeconds=60
> Apr 04, 2014 3:49:47 PM org.apache.solr.handler.XmlUpdateRequestHandler
> init
> WARNING: Using deprecated class: XmlUpdateRequestHandler -- replace with
> UpdateRequestHandler
> Apr 04, 2014 3:49:47 PM org.apache.solr.search.SolrIndexSearcher <init>
> INFO: Opening Searcher@35f79e13 main
> Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
> INFO: Hard AutoCommit: disabled
> Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
> INFO: Soft AutoCommit: disabled
> Apr 04, 2014 3:49:47 PM
> org.apache.solr.handler.component.SpellCheckComponent inform
> INFO: Initializing spell checkers
> Apr 04, 2014 3:49:47 PM
> org.apache.solr.handler.component.SpellCheckComponent inform
> INFO: No queryConverter defined, using default converter
> Apr 04, 2014 3:49:47 PM
> org.apache.solr.handler.component.QueryElevationComponent inform
> INFO: Loading QueryElevation from: D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\conf\elevate.xml
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.CoreContainer registerCore
> INFO: registering core: core0
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener sending requests to Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
> Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: user.dir=D:\AbcData\V12\application server\server\indexer\example
> Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init() done
> 2014-04-04 15:49:47.722:INFO:oejs.AbstractConnector:Started
> SocketConnector@0.0.0.0:8983
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore execute
> INFO: [core0] webapp=null path=null
> params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false}
> hits=0 status=0 QTime=28
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener done.
> Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore registerSearcher
> INFO: [core0] Registered new searcher Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
> maybeReloadConfiguration
> INFO: Loading DIH Configuration: database.xml
> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> INFO: Starting Full Import
> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrCore execute
> INFO: [core0] webapp=/solr path=/dataimport
> params={commit=true&command=full-import} status=0 QTime=12
> Apr 04, 2014 3:50:00 PM
> org.apache.solr.handler.dataimport.SimplePropertiesWriter
> readIndexerProperties
> INFO: Read dataimport.properties
> Apr 04, 2014 3:50:00 PM org.apache.solr.update.DirectUpdateHandler2
> deleteAll
> INFO: [core0] REMOVING ALL DOCUMENTS FROM INDEX
> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
> Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1[segments_1]
> Apr 04, 2014 3:50:00 PM
> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Creating a connection for entity full-index with URL:
> jdbc:oracle:thin:@test.abcdata.com:1521:ORCL
> Apr 04, 2014 3:50:00 PM
> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Time taken for getConnection(): 702
> MDFilter.class:::Message inserted for testing synch between compilation
> and running application (Roberto Corona: 05/31/2013 4:02pm
> parsepool.getNumActive(): 0
> Parsepool.getNumIdle(): 0
> creating new mdParse object
> Apr 04, 2014 3:50:17 PM org.apache.solr.core.SolrCore execute
> INFO: [core0] webapp=/solr path=/dataimport
> params={commit=true&command=full-import} status=0 QTime=0
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid97000.hprof ...
> Heap dump file created [62540489 bytes in 0.169 secs]
> Apr 04, 2014 3:50:26 PM
> org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [core0] webapp=/solr path=/dataimport
> params={commit=true&command=full-import} status=0 QTime=12
> {deleteByQuery=*:*,add=[1-5S, 1-8G, 1+DH+740, 1+DH+741, 1+DH+744, 1+DH+745,
> 1+DH+747, 1+DH+749, 1+DH+752, 1+DH+753, ... (4514 adds)]} 0 12
> Apr 04, 2014 3:50:26 PM org.apache.solr.common.SolrException log
> SEVERE: Full Import failed:java.lang.RuntimeException:
> java.lang.RuntimeException:
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:266)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
> Caused by: java.lang.RuntimeException:
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:406)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)
> ... 3 more
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
> ... 5 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator.getByteBlock(ByteBlockPool.java:103)
> at
> org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:202)
> at
> org.apache.lucene.util.ByteBlockPool.allocSlice(ByteBlockPool.java:257)
> at
> org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:252)
> at
> org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:273)
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:145)
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:227)
> at
> org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:235)
> at
> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:165)
> at
> org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:254)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:256)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:201)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:477)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
> at
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
> at
> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
> at
> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:235)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:500)
> ... 6 more
>
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: start rollback{}
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
> newIndexWriter
> INFO: Creating new IndexWriter...
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
> newIndexWriter
> INFO: Waiting until IndexWriter is unused... core=core0
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
> newIndexWriter
> INFO: Rollback old IndexWriter... core=core0
> Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=D:\AbcData\V12\application
> server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
> Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1[segments_1]
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
> newIndexWriter
> INFO: New IndexWriter is ready to be used.
> Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: end_rollback
>
>
>
> On Fri, Apr 4, 2014 at 4:59 PM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
>> Guessing that the attachments won't work, I am pasting one file in each
>> of four separate emails.
>>
>> database.xml
>>
>>
>> <dataConfig>
>> <dataSource
>> name="org_only"
>> type="JdbcDataSource"
>> driver="oracle.jdbc.OracleDriver"
>> url="jdbc:oracle:thin:@test.abcdata.com:1521:ORCL"
>> user="admin"
>> password="admin"
>> readOnly="false"
>> />
>> <document>
>>
>>
>> <entity name="full-index" query="
>> select
>>
>> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(100)), 'null')
>> as SOLR_ID,
>>
>> 'ORACLE.ADDRESS_ALL'
>> as SOLR_CATEGORY,
>>
>> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(255)), ' ') as
>> ADDRESSALLROWID,
>> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>> ADDRESSALLADDRTYPECD,
>> NVL(cast(ORACLE.ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLONGITUDE,
>> NVL(cast(ORACLE.ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLATITUDE,
>> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
>> ADDRESSALLADDRNAME,
>> NVL(cast(ORACLE.ADDRESS_ALL.CITY as varchar2(255)), ' ') as
>> ADDRESSALLCITY,
>> NVL(cast(ORACLE.ADDRESS_ALL.STATE as varchar2(255)), ' ') as
>> ADDRESSALLSTATE,
>> NVL(cast(ORACLE.ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>> ADDRESSALLEMAILADDR
>>
>> from ORACLE.ADDRESS_ALL
>> " >
>>
>> <field column="SOLR_ID" name="id" />
>> <field column="SOLR_CATEGORY" name="category" />
>> <field column="ADDRESSALLROWID" name="ADDRESS_ALL.RECORD_ID_abc" />
>> <field column="ADDRESSALLADDRTYPECD" name="ADDRESS_ALL.ADDR_TYPE_CD_abc"
>> />
>> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ALL.LONGITUDE_abc" />
>> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ALL.LATITUDE_abc" />
>> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ALL.ADDR_NAME_abc" />
>> <field column="ADDRESSALLCITY" name="ADDRESS_ALL.CITY_abc" />
>> <field column="ADDRESSALLSTATE" name="ADDRESS_ALL.STATE_abc" />
>> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ALL.EMAIL_ADDR_abc" />
>>
>> </entity>
>>
>>
>>
>> <!-- Varaibles -->
>> <!-- '${dataimporter.last_index_time}' -->
>> </document>
>> </dataConfig>
>>
>>
>>
>> On Fri, Apr 4, 2014 at 4:57 PM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>>> Does this user list allow attachments? I have four files attached
>>> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
>>> process again using the parameters you suggested, but not to a csv file.
>>> It errored out quickly. We are working on the csv file run.
>>>
>>> Removed both <autoCommit> and <autoSoftCommit> parts/definitions from
>>> solrconfig.xml
>>>
>>> Disabled tlog by removing
>>>
>>> <updateLog>
>>> <str name="dir">${solr.ulog.dir:}</str>
>>> </updateLog>
>>>
>>> from solrconfig.xml
>>>
>>> Used commit=true parameter. ?commit=true&command=full-import
>>>
>>>
>>> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This may not solve your problem but generally it is recommended to
>>>> disable auto commit and transaction logs for bulk indexing.
>>>> And issue one commit at the very end. Do you tlogs enabled? I see
>>>> "commit failed" in the error message thats why I am offering this.
>>>>
>>>> And regarding comma separated values, with this approach you focus on
>>>> just solr importing process. You separate data acquisition phrase. And it
>>>> is very fast load even big csv files
>>>> http://wiki.apache.org/solr/UpdateCSV
>>>> I have never experienced OOM during indexing, I suspect data
>>>> acquisition has role in it.
>>>>
>>>> Ahmet
>>>>
>>>> On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
>>>> candygram.for.mongo@gmail.com> wrote:
>>>>
>>>> We would be happy to try that. That sounds counter intuitive for the
>>>> high volume of records we have. Can you help me understand how that might
>>>> solve our problem?
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>>
>>>> Hi,
>>>> >
>>>> >Can you remove auto commit for bulk import. Commit at the very end?
>>>> >
>>>> >Ahmet
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>>>> candygram.for.mongo@gmail.com> wrote:
>>>> >In case the attached database.xml file didn't show up, I have pasted
>>>> in the
>>>> >contents below:
>>>> >
>>>> ><dataConfig>
>>>> ><dataSource
>>>> >name="org_only"
>>>> >type="JdbcDataSource"
>>>> >driver="oracle.jdbc.OracleDriver"
>>>> >url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>>>> >user="admin"
>>>> >password="admin"
>>>> >readOnly="false"
>>>> >batchSize="100"
>>>> >/>
>>>> ><document>
>>>> >
>>>> >
>>>> ><entity name="full-index" query="
>>>> >select
>>>> >
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>>>> >as SOLR_ID,
>>>> >
>>>> >'ORCL.ADDRESS_ACCT_ALL'
>>>> >as SOLR_CATEGORY,
>>>> >
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>>>> >ADDRESSALLROWID,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>>>> >ADDRESSALLADDRTYPECD,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>>>> >ADDRESSALLLONGITUDE,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>>>> >ADDRESSALLLATITUDE,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>>>> >ADDRESSALLADDRNAME,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>>>> >ADDRESSALLCITY,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>>>> >ADDRESSALLSTATE,
>>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>>>> >ADDRESSALLEMAILADDR
>>>> >
>>>> >from ORCL.ADDRESS_ACCT_ALL
>>>> >" >
>>>> >
>>>> ><field column="SOLR_ID" name="id" />
>>>> ><field column="SOLR_CATEGORY" name="category" />
>>>> ><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc"
>>>> />
>>>> ><field column="ADDRESSALLADDRTYPECD"
>>>> >name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>>>> ><field column="ADDRESSALLLONGITUDE"
>>>> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
>>>> ><field column="ADDRESSALLLATITUDE"
>>>> name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
>>>> ><field column="ADDRESSALLADDRNAME"
>>>> name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
>>>> ><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>>>> ><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>>>> ><field column="ADDRESSALLEMAILADDR"
>>>> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>>>> >/>
>>>> >
>>>> ></entity>
>>>> >
>>>> >
>>>> >
>>>> ><!-- Varaibles -->
>>>> ><!-- '${dataimporter.last_index_time}' -->
>>>> ></document>
>>>> ></dataConfig>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>>>> >candygram.for.mongo@gmail.com> wrote:
>>>> >
>>>> >> In this case we are indexing an Oracle database.
>>>> >>
>>>> >> We do not include the data-config.xml in our distribution. We store
>>>> the
>>>> >> database information in the database.xml file. I have attached the
>>>> >> database.xml file.
>>>> >>
>>>> >> When we use the default merge policy settings, we get the same
>>>> results.
>>>> >>
>>>> >>
>>>> >>
>>>> >> We have not tried to dump the table to a comma separated file. We
>>>> think
>>>> >> that dumping this size table to disk will introduce other memory
>>>> problems
>>>> >> with big file management. We have not tested that case.
>>>> >>
>>>> >>
>>>> >> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
>>>> wrote:
>>>> >>
>>>> >>> Hi,
>>>> >>>
>>>> >>> Which database are you using? Can you send us data-config.xml?
>>>> >>>
>>>> >>> What happens when you use default merge policy settings?
>>>> >>>
>>>> >>> What happens when you dump your table to Comma Separated File and
>>>> fed
>>>> >>> that file to solr?
>>>> >>>
>>>> >>> Ahmet
>>>> >>>
>>>> >>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>>>> >>> candygram.for.mongo@gmail.com> wrote:
>>>> >>>
>>>> >>> The ramBufferSizeMB was set to 6MB only on the test system to make
>>>> the
>>>> >>> system crash sooner. In production that tag is commented out which
>>>> >>> I believe forces the default value to be used.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>>>> wrote:
>>>> >>>
>>>> >>> Hi,
>>>> >>> >
>>>> >>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>>>> >>> >
>>>> >>> >Ahmet
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>>>> >>> candygram.for.mongo@gmail.com> wrote:
>>>> >>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>>>> Exception
>>>> >>> >
>>>> >>> >*SOLR/Lucene version: *4.2.1*
>>>> >>> >
>>>> >>> >
>>>> >>> >*JVM version:
>>>> >>> >
>>>> >>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>>>> >>> >
>>>> >>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >*Indexer startup command:
>>>> >>> >
>>>> >>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >java " %JVMARGS% ^
>>>> >>> >
>>>> >>> >-Dcom.sun.management.jmxremote.port=1092 ^
>>>> >>> >
>>>> >>> >-Dcom.sun.management.jmxremote.ssl=false ^
>>>> >>> >
>>>> >>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>>>> >>> >
>>>> >>> >-jar start.jar
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >*SOLR indexing HTTP parameters request:
>>>> >>> >
>>>> >>> >webapp=/solr path=/dataimport
>>>> >>> >params={clean=false&command=full-import&wt=javabin&version=2}
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >We are getting a Java heap OOM exception when indexing (updating)
>>>> 27
>>>> >>> >million records. If we increase the Java heap memory settings the
>>>> >>> problem
>>>> >>> >goes away but we believe the problem has not been fixed and that
>>>> we will
>>>> >>> >eventually get the same OOM exception. We have other processes on
>>>> the
>>>> >>> >server that also require resources so we cannot continually
>>>> increase the
>>>> >>> >memory settings to resolve the OOM issue. We are trying to find a
>>>> way to
>>>> >>> >configure the SOLR instance to reduce or preferably eliminate the
>>>> >>> >possibility of an OOM exception.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >We can reproduce the problem on a test machine. We set the Java
>>>> heap
>>>> >>> >memory size to 64MB to accelerate the exception. If we increase
>>>> this
>>>> >>> >setting the same problems occurs, just hours later. In the test
>>>> >>> >environment, we are using the following parameters:
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Normally we use the default solrconfig.xml file with only the
>>>> following
>>>> >>> jar
>>>> >>> >file references added:
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> ><lib path="../../../../default/lib/common.jar" />
>>>> >>> >
>>>> >>> ><lib path="../../../../default/lib/webapp.jar" />
>>>> >>> >
>>>> >>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Using these values and trying to index 6 million records from the
>>>> >>> database,
>>>> >>> >the Java Heap Out of Memory exception is thrown very quickly.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >We were able to complete a successful indexing by further
>>>> modifying the
>>>> >>> >solrconfig.xml and removing all or all but one <copyfield> tags
>>>> from the
>>>> >>> >schema.xml file.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >The following solrconfig.xml values were modified:
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>>> >>> >
>>>> >>> ><int name="maxMergeAtOnce">2</int>
>>>> >>> >
>>>> >>> ><int name="maxMergeAtOnceExplicit">2</int>
>>>> >>> >
>>>> >>> ><int name="segmentsPerTier">10</int>
>>>> >>> >
>>>> >>> ><int name="maxMergedSegmentMB">150</int>
>>>> >>> >
>>>> >>> ></mergePolicy>
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> ><autoCommit>
>>>> >>> >
>>>> >>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before
>>>> this -- >
>>>> >>> >
>>>> >>> ><openSearcher>false</openSearcher>
>>>> >>> >
>>>> >>> ></autoCommit>
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Using our customized schema.xml file with two or more <copyfield>
>>>> tags,
>>>> >>> the
>>>> >>> >OOM exception is always thrown. Based on the errors, the problem
>>>> occurs
>>>> >>> >when the process was trying to do the merge. The error is provided
>>>> >>> below:
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Exception in thread "Lucene Merge Thread #156"
>>>> >>> >org.apache.lucene.index.MergePolicy$MergeException:
>>>> >>> >java.lang.OutOfMemoryError: Java heap space
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>>> >>> >
>>>> >>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>>>> >>> >
>>>> >>> > at
>>>> >>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>>>> >>> >
>>>> >>> > at
>>>> >>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>>> >>> >
>>>> >>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>>>> >>> >
>>>> >>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
>>>> writer
>>>> >>> >hit an OutOfMemoryError; cannot commit
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>>>> >>> >
>>>> >>> > at
>>>> >>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>>>> >>> >
>>>> >>> > at
>>>> >>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> >>> >
>>>> >>> > at
>>>> >>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>> >>> >
>>>> >>> > at
>>>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> >>> >
>>>> >>> > at
>>>> >>>
>>>> >>>
>>>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> >>> >
>>>> >>> > at java.lang.Thread.run(Thread.java:722)
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >We think but are not 100% sure that the problem is related to the
>>>> merge.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Normally our schema.xml contains a lot of field specifications
>>>> (like the
>>>> >>> >ones seen in the file fragment below):
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> >>> dest="ADDRESS.RECORD_ID.case_abc"
>>>> >>> >/>
>>>> >>> >
>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> >>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>>>> >>> >
>>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> >>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >In tests using the default file schema.xml and no <copyfield> tags,
>>>> >>> >indexing completed successfully. 6 million records produced a 900
>>>> MB
>>>> >>> data
>>>> >>> >directory.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >When I included just one <copyfield> tag, indexing completed
>>>> >>> successfully. 6
>>>> >>> >million records produced a 990 MB data directory (90 MB bigger).
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >When I included just two <copyfield> tags, the index crashed with
>>>> an OOM
>>>> >>> >exception.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
>>>> postponed
>>>> >>> the
>>>> >>> >crash.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >The net of our test results I as follows:
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >*solrconfig.xml*
>>>> >>> >
>>>> >>> >*schema.xml*
>>>> >>> >
>>>> >>> >*result*
>>>> >>> >
>>>> >>> >
>>>> >>> >default plus only jar references
>>>> >>> >
>>>> >>> >default (no copyfield tags)
>>>> >>> >
>>>> >>> >success
>>>> >>> >
>>>> >>> >default plus only jar references
>>>> >>> >
>>>> >>> >modified with one copyfield tag
>>>> >>> >
>>>> >>> >success
>>>> >>> >
>>>> >>> >default plus only jar references
>>>> >>> >
>>>> >>> >modified with two copyfield tags
>>>> >>> >
>>>> >>> >crash
>>>> >>> >
>>>> >>> >additional modified settings
>>>> >>> >
>>>> >>> >default (no copyfield tags)
>>>> >>> >
>>>> >>> >success
>>>> >>> >
>>>> >>> >additional modified settings
>>>> >>> >
>>>> >>> >modified with one copyfield tag
>>>> >>> >
>>>> >>> >success
>>>> >>> >
>>>> >>> >additional modified settings
>>>> >>> >
>>>> >>> >modified with two copyfield tags
>>>> >>> >
>>>> >>> >crash
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >Our question is, what can we do to eliminate these OOM exceptions?
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
error.txt below
Java Platform Detected x64
Java Platform Detected -XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
-XX:+HeapDumpOnOutOfMemoryError -XX:+CreateMinidumpOnCrash
2014-04-04 15:49:43.341:INFO:oejs.Server:jetty-8.1.8.v20121106
2014-04-04 15:49:43.353:INFO:oejdp.ScanningAppProvider:Deployment monitor
D:\AbcData\V12\application server\server\indexer\example\contexts at
interval 0
2014-04-04 15:49:43.358:INFO:oejd.DeploymentManager:Deployable added:
D:\AbcData\V12\application
server\server\indexer\example\contexts\solr-jetty-context.xml
2014-04-04 15:49:43.989:INFO:oejw.StandardDescriptorProcessor:NO JSP
Support for /solr, did not find org.apache.jasper.servlet.JspServlet
Null identity service, trying login service: null
Finding identity service: null
2014-04-04 15:49:44.011:INFO:oejsh.ContextHandler:started
o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
server\server\indexer\example/webapps/solr.war
2014-04-04 15:49:44.012:INFO:oejsh.ContextHandler:started
o.e.j.w.WebAppContext{/solr,file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr-webapp/webapp/},D:\AbcData\V12\application
server\server\indexer\example/webapps/solr.war
Apr 04, 2014 3:49:44 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
locateSolrHome
INFO: JNDI not configured for solr (NoInitialContextEx)
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
locateSolrHome
INFO: solr home defaulted to 'solr/' (could not find system property or
JNDI)
Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: D:\AbcData\V12\application
server\server\indexer\example\solr\solr.xml
Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer <init>
INFO: New CoreContainer 1879341237
Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
INFO: Loading CoreContainer using Solr Home: 'solr/'
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
INFO: new SolrResourceLoader for directory: 'solr/'
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/apache-log4j-extras-1.2.17.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/jtds-1.2.5.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/log4j-1.2.17.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msbase.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/mssqlserver.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/msutil.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/ojdbc6.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-api-1.7.5.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/slf4j-nop-1.7.5.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/example/solr/lib/sqljdbc4.jar'
to classloader
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting socketTimeout to: 0
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting urlScheme to: http://
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting connTimeout to: 0
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting maxConnectionsPerHost to: 20
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting corePoolSize to: 0
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting maximumPoolSize to: 2147483647
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting maxThreadIdleTime to: 5
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting sizeOfQueue to: -1
Apr 04, 2014 3:49:44 PM
org.apache.solr.handler.component.HttpShardHandlerFactory getParameter
INFO: Setting fairnessPolicy to: false
Apr 04, 2014 3:49:44 PM org.apache.solr.client.solrj.impl.HttpClientUtil
createClient
INFO: Creating new http client,
config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer load
INFO: Registering Log Listener
Apr 04, 2014 3:49:44 PM org.apache.solr.core.CoreContainer create
INFO: Creating SolrCore 'core0' using instanceDir: solr\core0
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader <init>
INFO: new SolrResourceLoader for directory: 'solr\core0\'
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig initLibs
INFO: Adding specified lib dirs to ClassLoader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-4.2.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/dist/solr-dataimporthandler-extras-4.2.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-core-0.7.2.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/apache-mime4j-dom-0.7.2.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcmail-jdk15-1.45.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/bcprov-jdk15-1.45.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/boilerpipe-1.1.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/commons-compress-1.4.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/dom4j-1.6.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/fontbox-1.7.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/icu4j-49.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/isoparser-1.0-RC-1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jdom-1.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/jempbox-1.7.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/juniversalchardet-1.0.3.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/metadata-extractor-2.6.2.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/netcdf-4.2-min.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/pdfbox-1.7.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-3.8.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-3.8.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-ooxml-schemas-3.8.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/poi-scratchpad-3.8.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/rome-0.9.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tagsoup-1.2.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-core-1.3.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/tika-parsers-1.3.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-core-0.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/vorbis-java-tika-0.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xercesImpl-2.9.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xmlbeans-2.3.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/extraction/lib/xz-1.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/attributes-binder-1.0.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/carrot2-mini-3.6.2.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/hppc-0.4.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-core-asl-1.7.4.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/jackson-mapper-asl-1.7.4.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-collections-1.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/mahout-math-0.6.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/clustering/lib/simple-xml-2.6.4.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/jsonic-1.2.7.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/langid/lib/langdetect-1.1-20120112.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-beanutils-1.7.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/commons-collections-3.2.1.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-1.7.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/indexer/contrib/velocity/lib/velocity-tools-2.0.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/default/lib/common.jar' to
classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/default/lib/webapp.jar' to
classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding
'file:/D:/AbcData/V12/applicationserver/server/default/lib/commons-pool-1.4.jar'
to classloader
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
INFO: Using Lucene MatchVersion: LUCENE_42
Apr 04, 2014 3:49:44 PM org.apache.solr.core.SolrConfig <init>
INFO: Loaded SolrConfig: solrconfig.xml
Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
INFO: Reading Solr Schema
Apr 04, 2014 3:49:44 PM org.apache.solr.schema.IndexSchema readSchema
INFO: Schema name=example
Apr 04, 2014 3:49:46 PM org.apache.solr.schema.IndexSchema readSchema
INFO: unique key field: id
Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
reload
INFO: Reloading exchange rates from file currency.xml
Apr 04, 2014 3:49:47 PM org.apache.solr.schema.FileExchangeRateProvider
reload
INFO: Reloading exchange rates from file currency.xml
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
INFO: org.apache.solr.core.StandardDirectoryFactory
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
INFO: [core0] Opening new SolrCore at solr\core0\,
dataDir=D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore <init>
INFO: JMX monitoring not detected for core: core0
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
INFO: [core0] Added SolrEventListener for newSearcher:
org.apache.solr.core.QuerySenderListener{queries=[]}
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initListeners
INFO: [core0] Added SolrEventListener for firstSearcher:
org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher
warming in solrconfig.xml}]}
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initDirectoryFactory
INFO: org.apache.solr.core.StandardDirectoryFactory
Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
INFO: return new directory for D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data forceNew: false
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore getNewIndexDir
INFO: New index directory detected: old=null new=D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index/
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initIndex
WARNING: [core0] Solr index directory 'D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index' doesn't exist.
Creating new index...
Apr 04, 2014 3:49:47 PM org.apache.solr.core.CachingDirectoryFactory get
INFO: return new directory for D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index forceNew: false
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=1
commit{dir=D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1[segments_1]
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
INFO: created json: solr.JSONResponseWriter
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
INFO: adding lazy queryResponseWriter: solr.VelocityResponseWriter
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
INFO: created velocity: solr.VelocityResponseWriter
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore initWriters
INFO: created xslt: solr.XSLTResponseWriter
Apr 04, 2014 3:49:47 PM org.apache.solr.response.XSLTResponseWriter init
INFO: xsltCacheLifetimeSeconds=5
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /dataimport:
org.apache.solr.handler.dataimport.DataImportHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /select: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /query: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /browse: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /update: solr.XmlUpdateRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.extraction.ExtractingRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /update/extract: solr.extraction.ExtractingRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.FieldAnalysisRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /analysis/field: solr.FieldAnalysisRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /analysis/document: solr.DocumentAnalysisRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /admin/: solr.admin.AdminHandlers
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /admin/ping: solr.PingRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /debug/dump: solr.DumpRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.ReplicationHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /replication: solr.ReplicationHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /spell: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /tvrh: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /terms: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: adding lazy requestHandler: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.core.RequestHandlers
initHandlersFromConfig
INFO: created /elevate: solr.SearchHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.handler.loader.XMLLoader init
INFO: xsltCacheLifetimeSeconds=60
Apr 04, 2014 3:49:47 PM org.apache.solr.handler.XmlUpdateRequestHandler init
WARNING: Using deprecated class: XmlUpdateRequestHandler -- replace with
UpdateRequestHandler
Apr 04, 2014 3:49:47 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@35f79e13 main
Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
INFO: Hard AutoCommit: disabled
Apr 04, 2014 3:49:47 PM org.apache.solr.update.CommitTracker <init>
INFO: Soft AutoCommit: disabled
Apr 04, 2014 3:49:47 PM
org.apache.solr.handler.component.SpellCheckComponent inform
INFO: Initializing spell checkers
Apr 04, 2014 3:49:47 PM
org.apache.solr.handler.component.SpellCheckComponent inform
INFO: No queryConverter defined, using default converter
Apr 04, 2014 3:49:47 PM
org.apache.solr.handler.component.QueryElevationComponent inform
INFO: Loading QueryElevation from: D:\AbcData\V12\application
server\server\indexer\example\solr\core0\conf\elevate.xml
Apr 04, 2014 3:49:47 PM org.apache.solr.core.CoreContainer registerCore
INFO: registering core: core0
Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to
Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: user.dir=D:\AbcData\V12\application server\server\indexer\example
Apr 04, 2014 3:49:47 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init() done
2014-04-04 15:49:47.722:INFO:oejs.AbstractConnector:Started
SocketConnector@0.0.0.0:8983
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=null path=null
params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false}
hits=0 status=0 QTime=28
Apr 04, 2014 3:49:47 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Apr 04, 2014 3:49:47 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [core0] Registered new searcher
Searcher@35f79e13main{StandardDirectoryReader(segments_1:1)}
Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
maybeReloadConfiguration
INFO: Loading DIH Configuration: database.xml
Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
INFO: Data Configuration loaded successfully
Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=/solr path=/dataimport
params={commit=true&command=full-import} status=0 QTime=12
Apr 04, 2014 3:50:00 PM
org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
INFO: Read dataimport.properties
Apr 04, 2014 3:50:00 PM org.apache.solr.update.DirectUpdateHandler2
deleteAll
INFO: [core0] REMOVING ALL DOCUMENTS FROM INDEX
Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
Apr 04, 2014 3:50:00 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1[segments_1]
Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity full-index with URL:
jdbc:oracle:thin:@test.abcdata.com:1521:ORCL
Apr 04, 2014 3:50:00 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 702
MDFilter.class:::Message inserted for testing synch between compilation
and running application (Roberto Corona: 05/31/2013 4:02pm
parsepool.getNumActive(): 0
Parsepool.getNumIdle(): 0
creating new mdParse object
Apr 04, 2014 3:50:17 PM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=/solr path=/dataimport
params={commit=true&command=full-import} status=0 QTime=0
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid97000.hprof ...
Heap dump file created [62540489 bytes in 0.169 secs]
Apr 04, 2014 3:50:26 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: [core0] webapp=/solr path=/dataimport
params={commit=true&command=full-import} status=0 QTime=12
{deleteByQuery=*:*,add=[1-5S, 1-8G, 1+DH+740, 1+DH+741, 1+DH+744, 1+DH+745,
1+DH+747, 1+DH+749, 1+DH+752, 1+DH+753, ... (4514 adds)]} 0 12
Apr 04, 2014 3:50:26 PM org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:266)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:406)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)
... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
... 5 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator.getByteBlock(ByteBlockPool.java:103)
at
org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:202)
at
org.apache.lucene.util.ByteBlockPool.allocSlice(ByteBlockPool.java:257)
at
org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:252)
at
org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:273)
at
org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:145)
at
org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:227)
at
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:235)
at
org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:165)
at
org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:254)
at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:256)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:201)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:477)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:70)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:235)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:500)
... 6 more
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback{}
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Creating new IndexWriter...
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Waiting until IndexWriter is unused... core=core0
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Rollback old IndexWriter... core=core0
Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=D:\AbcData\V12\application
server\server\indexer\example\solr\core0\data\index,segFN=segments_1,generation=1,filenames=[segments_1]
Apr 04, 2014 3:50:26 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1[segments_1]
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: New IndexWriter is ready to be used.
Apr 04, 2014 3:50:26 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
On Fri, Apr 4, 2014 at 4:59 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> Guessing that the attachments won't work, I am pasting one file in each of
> four separate emails.
>
> database.xml
>
>
> <dataConfig>
> <dataSource
> name="org_only"
> type="JdbcDataSource"
> driver="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@test.abcdata.com:1521:ORCL"
> user="admin"
> password="admin"
> readOnly="false"
> />
> <document>
>
>
> <entity name="full-index" query="
> select
>
> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(100)), 'null')
> as SOLR_ID,
>
> 'ORACLE.ADDRESS_ALL'
> as SOLR_CATEGORY,
>
> NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(255)), ' ') as
> ADDRESSALLROWID,
> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> ADDRESSALLADDRTYPECD,
> NVL(cast(ORACLE.ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
> ADDRESSALLLONGITUDE,
> NVL(cast(ORACLE.ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
> ADDRESSALLLATITUDE,
> NVL(cast(ORACLE.ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
> ADDRESSALLADDRNAME,
> NVL(cast(ORACLE.ADDRESS_ALL.CITY as varchar2(255)), ' ') as ADDRESSALLCITY,
> NVL(cast(ORACLE.ADDRESS_ALL.STATE as varchar2(255)), ' ') as
> ADDRESSALLSTATE,
> NVL(cast(ORACLE.ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> ADDRESSALLEMAILADDR
>
> from ORACLE.ADDRESS_ALL
> " >
>
> <field column="SOLR_ID" name="id" />
> <field column="SOLR_CATEGORY" name="category" />
> <field column="ADDRESSALLROWID" name="ADDRESS_ALL.RECORD_ID_abc" />
> <field column="ADDRESSALLADDRTYPECD" name="ADDRESS_ALL.ADDR_TYPE_CD_abc" />
> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ALL.LONGITUDE_abc" />
> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ALL.LATITUDE_abc" />
> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ALL.ADDR_NAME_abc" />
> <field column="ADDRESSALLCITY" name="ADDRESS_ALL.CITY_abc" />
> <field column="ADDRESSALLSTATE" name="ADDRESS_ALL.STATE_abc" />
> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ALL.EMAIL_ADDR_abc" />
>
> </entity>
>
>
>
> <!-- Varaibles -->
> <!-- '${dataimporter.last_index_time}' -->
> </document>
> </dataConfig>
>
>
>
> On Fri, Apr 4, 2014 at 4:57 PM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
>> Does this user list allow attachments? I have four files attached
>> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
>> process again using the parameters you suggested, but not to a csv file.
>> It errored out quickly. We are working on the csv file run.
>>
>> Removed both <autoCommit> and <autoSoftCommit> parts/definitions from
>> solrconfig.xml
>>
>> Disabled tlog by removing
>>
>> <updateLog>
>> <str name="dir">${solr.ulog.dir:}</str>
>> </updateLog>
>>
>> from solrconfig.xml
>>
>> Used commit=true parameter. ?commit=true&command=full-import
>>
>>
>> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>
>>> Hi,
>>>
>>> This may not solve your problem but generally it is recommended to
>>> disable auto commit and transaction logs for bulk indexing.
>>> And issue one commit at the very end. Do you tlogs enabled? I see
>>> "commit failed" in the error message thats why I am offering this.
>>>
>>> And regarding comma separated values, with this approach you focus on
>>> just solr importing process. You separate data acquisition phrase. And it
>>> is very fast load even big csv files
>>> http://wiki.apache.org/solr/UpdateCSV
>>> I have never experienced OOM during indexing, I suspect data acquisition
>>> has role in it.
>>>
>>> Ahmet
>>>
>>> On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
>>> candygram.for.mongo@gmail.com> wrote:
>>>
>>> We would be happy to try that. That sounds counter intuitive for the
>>> high volume of records we have. Can you help me understand how that might
>>> solve our problem?
>>>
>>>
>>>
>>>
>>> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>
>>> Hi,
>>> >
>>> >Can you remove auto commit for bulk import. Commit at the very end?
>>> >
>>> >Ahmet
>>> >
>>> >
>>> >
>>> >
>>> >On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>>> candygram.for.mongo@gmail.com> wrote:
>>> >In case the attached database.xml file didn't show up, I have pasted in
>>> the
>>> >contents below:
>>> >
>>> ><dataConfig>
>>> ><dataSource
>>> >name="org_only"
>>> >type="JdbcDataSource"
>>> >driver="oracle.jdbc.OracleDriver"
>>> >url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>>> >user="admin"
>>> >password="admin"
>>> >readOnly="false"
>>> >batchSize="100"
>>> >/>
>>> ><document>
>>> >
>>> >
>>> ><entity name="full-index" query="
>>> >select
>>> >
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>>> >as SOLR_ID,
>>> >
>>> >'ORCL.ADDRESS_ACCT_ALL'
>>> >as SOLR_CATEGORY,
>>> >
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>>> >ADDRESSALLROWID,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>>> >ADDRESSALLADDRTYPECD,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>>> >ADDRESSALLLONGITUDE,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>>> >ADDRESSALLLATITUDE,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>>> >ADDRESSALLADDRNAME,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>>> >ADDRESSALLCITY,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>>> >ADDRESSALLSTATE,
>>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>>> >ADDRESSALLEMAILADDR
>>> >
>>> >from ORCL.ADDRESS_ACCT_ALL
>>> >" >
>>> >
>>> ><field column="SOLR_ID" name="id" />
>>> ><field column="SOLR_CATEGORY" name="category" />
>>> ><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
>>> ><field column="ADDRESSALLADDRTYPECD"
>>> >name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>>> ><field column="ADDRESSALLLONGITUDE"
>>> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
>>> ><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc"
>>> />
>>> ><field column="ADDRESSALLADDRNAME"
>>> name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
>>> ><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>>> ><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>>> ><field column="ADDRESSALLEMAILADDR"
>>> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>>> >/>
>>> >
>>> ></entity>
>>> >
>>> >
>>> >
>>> ><!-- Varaibles -->
>>> ><!-- '${dataimporter.last_index_time}' -->
>>> ></document>
>>> ></dataConfig>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>>> >candygram.for.mongo@gmail.com> wrote:
>>> >
>>> >> In this case we are indexing an Oracle database.
>>> >>
>>> >> We do not include the data-config.xml in our distribution. We store
>>> the
>>> >> database information in the database.xml file. I have attached the
>>> >> database.xml file.
>>> >>
>>> >> When we use the default merge policy settings, we get the same
>>> results.
>>> >>
>>> >>
>>> >>
>>> >> We have not tried to dump the table to a comma separated file. We
>>> think
>>> >> that dumping this size table to disk will introduce other memory
>>> problems
>>> >> with big file management. We have not tested that case.
>>> >>
>>> >>
>>> >> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
>>> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> Which database are you using? Can you send us data-config.xml?
>>> >>>
>>> >>> What happens when you use default merge policy settings?
>>> >>>
>>> >>> What happens when you dump your table to Comma Separated File and fed
>>> >>> that file to solr?
>>> >>>
>>> >>> Ahmet
>>> >>>
>>> >>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>>> >>> candygram.for.mongo@gmail.com> wrote:
>>> >>>
>>> >>> The ramBufferSizeMB was set to 6MB only on the test system to make
>>> the
>>> >>> system crash sooner. In production that tag is commented out which
>>> >>> I believe forces the default value to be used.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>>> wrote:
>>> >>>
>>> >>> Hi,
>>> >>> >
>>> >>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>>> >>> >
>>> >>> >Ahmet
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>>> >>> candygram.for.mongo@gmail.com> wrote:
>>> >>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>>> Exception
>>> >>> >
>>> >>> >*SOLR/Lucene version: *4.2.1*
>>> >>> >
>>> >>> >
>>> >>> >*JVM version:
>>> >>> >
>>> >>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>>> >>> >
>>> >>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >*Indexer startup command:
>>> >>> >
>>> >>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >java " %JVMARGS% ^
>>> >>> >
>>> >>> >-Dcom.sun.management.jmxremote.port=1092 ^
>>> >>> >
>>> >>> >-Dcom.sun.management.jmxremote.ssl=false ^
>>> >>> >
>>> >>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>>> >>> >
>>> >>> >-jar start.jar
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >*SOLR indexing HTTP parameters request:
>>> >>> >
>>> >>> >webapp=/solr path=/dataimport
>>> >>> >params={clean=false&command=full-import&wt=javabin&version=2}
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >We are getting a Java heap OOM exception when indexing (updating) 27
>>> >>> >million records. If we increase the Java heap memory settings the
>>> >>> problem
>>> >>> >goes away but we believe the problem has not been fixed and that we
>>> will
>>> >>> >eventually get the same OOM exception. We have other processes on
>>> the
>>> >>> >server that also require resources so we cannot continually
>>> increase the
>>> >>> >memory settings to resolve the OOM issue. We are trying to find a
>>> way to
>>> >>> >configure the SOLR instance to reduce or preferably eliminate the
>>> >>> >possibility of an OOM exception.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >We can reproduce the problem on a test machine. We set the Java
>>> heap
>>> >>> >memory size to 64MB to accelerate the exception. If we increase
>>> this
>>> >>> >setting the same problems occurs, just hours later. In the test
>>> >>> >environment, we are using the following parameters:
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Normally we use the default solrconfig.xml file with only the
>>> following
>>> >>> jar
>>> >>> >file references added:
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> ><lib path="../../../../default/lib/common.jar" />
>>> >>> >
>>> >>> ><lib path="../../../../default/lib/webapp.jar" />
>>> >>> >
>>> >>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Using these values and trying to index 6 million records from the
>>> >>> database,
>>> >>> >the Java Heap Out of Memory exception is thrown very quickly.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >We were able to complete a successful indexing by further modifying
>>> the
>>> >>> >solrconfig.xml and removing all or all but one <copyfield> tags
>>> from the
>>> >>> >schema.xml file.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >The following solrconfig.xml values were modified:
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>> >>> >
>>> >>> ><int name="maxMergeAtOnce">2</int>
>>> >>> >
>>> >>> ><int name="maxMergeAtOnceExplicit">2</int>
>>> >>> >
>>> >>> ><int name="segmentsPerTier">10</int>
>>> >>> >
>>> >>> ><int name="maxMergedSegmentMB">150</int>
>>> >>> >
>>> >>> ></mergePolicy>
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> ><autoCommit>
>>> >>> >
>>> >>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before
>>> this -- >
>>> >>> >
>>> >>> ><openSearcher>false</openSearcher>
>>> >>> >
>>> >>> ></autoCommit>
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Using our customized schema.xml file with two or more <copyfield>
>>> tags,
>>> >>> the
>>> >>> >OOM exception is always thrown. Based on the errors, the problem
>>> occurs
>>> >>> >when the process was trying to do the merge. The error is provided
>>> >>> below:
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Exception in thread "Lucene Merge Thread #156"
>>> >>> >org.apache.lucene.index.MergePolicy$MergeException:
>>> >>> >java.lang.OutOfMemoryError: Java heap space
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>> >>> >
>>> >>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>>> >>> >
>>> >>> > at
>>> >>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>>> >>> >
>>> >>> > at
>>> >>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>> >>> >
>>> >>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>>> >>> >
>>> >>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
>>> writer
>>> >>> >hit an OutOfMemoryError; cannot commit
>>> >>> >
>>> >>> > at
>>> >>>
>>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>>> >>> >
>>> >>> > at
>>> >>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>>> >>> >
>>> >>> > at
>>> >>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> >>> >
>>> >>> > at
>>> >>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >>> >
>>> >>> > at
>>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> >>> >
>>> >>> > at
>>> >>>
>>> >>>
>>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> >>> >
>>> >>> > at java.lang.Thread.run(Thread.java:722)
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >We think but are not 100% sure that the problem is related to the
>>> merge.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Normally our schema.xml contains a lot of field specifications
>>> (like the
>>> >>> >ones seen in the file fragment below):
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> >>> dest="ADDRESS.RECORD_ID.case_abc"
>>> >>> >/>
>>> >>> >
>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> >>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>>> >>> >
>>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> >>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >In tests using the default file schema.xml and no <copyfield> tags,
>>> >>> >indexing completed successfully. 6 million records produced a 900
>>> MB
>>> >>> data
>>> >>> >directory.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >When I included just one <copyfield> tag, indexing completed
>>> >>> successfully. 6
>>> >>> >million records produced a 990 MB data directory (90 MB bigger).
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >When I included just two <copyfield> tags, the index crashed with
>>> an OOM
>>> >>> >exception.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
>>> postponed
>>> >>> the
>>> >>> >crash.
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >The net of our test results I as follows:
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >*solrconfig.xml*
>>> >>> >
>>> >>> >*schema.xml*
>>> >>> >
>>> >>> >*result*
>>> >>> >
>>> >>> >
>>> >>> >default plus only jar references
>>> >>> >
>>> >>> >default (no copyfield tags)
>>> >>> >
>>> >>> >success
>>> >>> >
>>> >>> >default plus only jar references
>>> >>> >
>>> >>> >modified with one copyfield tag
>>> >>> >
>>> >>> >success
>>> >>> >
>>> >>> >default plus only jar references
>>> >>> >
>>> >>> >modified with two copyfield tags
>>> >>> >
>>> >>> >crash
>>> >>> >
>>> >>> >additional modified settings
>>> >>> >
>>> >>> >default (no copyfield tags)
>>> >>> >
>>> >>> >success
>>> >>> >
>>> >>> >additional modified settings
>>> >>> >
>>> >>> >modified with one copyfield tag
>>> >>> >
>>> >>> >success
>>> >>> >
>>> >>> >additional modified settings
>>> >>> >
>>> >>> >modified with two copyfield tags
>>> >>> >
>>> >>> >crash
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >Our question is, what can we do to eliminate these OOM exceptions?
>>> >>> >
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>> >
>>> >
>>>
>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
Guessing that the attachments won't work, I am pasting one file in each of
four separate emails.
database.xml
<dataConfig>
<dataSource
name="org_only"
type="JdbcDataSource"
driver="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@test.abcdata.com:1521:ORCL"
user="admin"
password="admin"
readOnly="false"
/>
<document>
<entity name="full-index" query="
select
NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(100)), 'null')
as SOLR_ID,
'ORACLE.ADDRESS_ALL'
as SOLR_CATEGORY,
NVL(cast(ORACLE.ADDRESS_ALL.RECORD_ID as varchar2(255)), ' ') as
ADDRESSALLROWID,
NVL(cast(ORACLE.ADDRESS_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
ADDRESSALLADDRTYPECD,
NVL(cast(ORACLE.ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
ADDRESSALLLONGITUDE,
NVL(cast(ORACLE.ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
ADDRESSALLLATITUDE,
NVL(cast(ORACLE.ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
ADDRESSALLADDRNAME,
NVL(cast(ORACLE.ADDRESS_ALL.CITY as varchar2(255)), ' ') as ADDRESSALLCITY,
NVL(cast(ORACLE.ADDRESS_ALL.STATE as varchar2(255)), ' ') as
ADDRESSALLSTATE,
NVL(cast(ORACLE.ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
ADDRESSALLEMAILADDR
from ORACLE.ADDRESS_ALL
" >
<field column="SOLR_ID" name="id" />
<field column="SOLR_CATEGORY" name="category" />
<field column="ADDRESSALLROWID" name="ADDRESS_ALL.RECORD_ID_abc" />
<field column="ADDRESSALLADDRTYPECD" name="ADDRESS_ALL.ADDR_TYPE_CD_abc" />
<field column="ADDRESSALLLONGITUDE" name="ADDRESS_ALL.LONGITUDE_abc" />
<field column="ADDRESSALLLATITUDE" name="ADDRESS_ALL.LATITUDE_abc" />
<field column="ADDRESSALLADDRNAME" name="ADDRESS_ALL.ADDR_NAME_abc" />
<field column="ADDRESSALLCITY" name="ADDRESS_ALL.CITY_abc" />
<field column="ADDRESSALLSTATE" name="ADDRESS_ALL.STATE_abc" />
<field column="ADDRESSALLEMAILADDR" name="ADDRESS_ALL.EMAIL_ADDR_abc" />
</entity>
<!-- Varaibles -->
<!-- '${dataimporter.last_index_time}' -->
</document>
</dataConfig>
On Fri, Apr 4, 2014 at 4:57 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> Does this user list allow attachments? I have four files attached
> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
> process again using the parameters you suggested, but not to a csv file.
> It errored out quickly. We are working on the csv file run.
>
> Removed both <autoCommit> and <autoSoftCommit> parts/definitions from
> solrconfig.xml
>
> Disabled tlog by removing
>
> <updateLog>
> <str name="dir">${solr.ulog.dir:}</str>
> </updateLog>
>
> from solrconfig.xml
>
> Used commit=true parameter. ?commit=true&command=full-import
>
>
> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Hi,
>>
>> This may not solve your problem but generally it is recommended to
>> disable auto commit and transaction logs for bulk indexing.
>> And issue one commit at the very end. Do you tlogs enabled? I see "commit
>> failed" in the error message thats why I am offering this.
>>
>> And regarding comma separated values, with this approach you focus on
>> just solr importing process. You separate data acquisition phrase. And it
>> is very fast load even big csv files
>> http://wiki.apache.org/solr/UpdateCSV
>> I have never experienced OOM during indexing, I suspect data acquisition
>> has role in it.
>>
>> Ahmet
>>
>> On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>> We would be happy to try that. That sounds counter intuitive for the
>> high volume of records we have. Can you help me understand how that might
>> solve our problem?
>>
>>
>>
>>
>> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>
>> Hi,
>> >
>> >Can you remove auto commit for bulk import. Commit at the very end?
>> >
>> >Ahmet
>> >
>> >
>> >
>> >
>> >On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>> >In case the attached database.xml file didn't show up, I have pasted in
>> the
>> >contents below:
>> >
>> ><dataConfig>
>> ><dataSource
>> >name="org_only"
>> >type="JdbcDataSource"
>> >driver="oracle.jdbc.OracleDriver"
>> >url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>> >user="admin"
>> >password="admin"
>> >readOnly="false"
>> >batchSize="100"
>> >/>
>> ><document>
>> >
>> >
>> ><entity name="full-index" query="
>> >select
>> >
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>> >as SOLR_ID,
>> >
>> >'ORCL.ADDRESS_ACCT_ALL'
>> >as SOLR_CATEGORY,
>> >
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>> >ADDRESSALLROWID,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>> >ADDRESSALLADDRTYPECD,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>> >ADDRESSALLLONGITUDE,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>> >ADDRESSALLLATITUDE,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>> >ADDRESSALLADDRNAME,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>> >ADDRESSALLCITY,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>> >ADDRESSALLSTATE,
>> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>> >ADDRESSALLEMAILADDR
>> >
>> >from ORCL.ADDRESS_ACCT_ALL
>> >" >
>> >
>> ><field column="SOLR_ID" name="id" />
>> ><field column="SOLR_CATEGORY" name="category" />
>> ><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
>> ><field column="ADDRESSALLADDRTYPECD"
>> >name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>> ><field column="ADDRESSALLLONGITUDE"
>> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
>> ><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc"
>> />
>> ><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
>> />
>> ><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>> ><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>> ><field column="ADDRESSALLEMAILADDR"
>> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>> >/>
>> >
>> ></entity>
>> >
>> >
>> >
>> ><!-- Varaibles -->
>> ><!-- '${dataimporter.last_index_time}' -->
>> ></document>
>> ></dataConfig>
>> >
>> >
>> >
>> >
>> >
>> >
>> >On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>> >candygram.for.mongo@gmail.com> wrote:
>> >
>> >> In this case we are indexing an Oracle database.
>> >>
>> >> We do not include the data-config.xml in our distribution. We store
>> the
>> >> database information in the database.xml file. I have attached the
>> >> database.xml file.
>> >>
>> >> When we use the default merge policy settings, we get the same results.
>> >>
>> >>
>> >>
>> >> We have not tried to dump the table to a comma separated file. We
>> think
>> >> that dumping this size table to disk will introduce other memory
>> problems
>> >> with big file management. We have not tested that case.
>> >>
>> >>
>> >> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Which database are you using? Can you send us data-config.xml?
>> >>>
>> >>> What happens when you use default merge policy settings?
>> >>>
>> >>> What happens when you dump your table to Comma Separated File and fed
>> >>> that file to solr?
>> >>>
>> >>> Ahmet
>> >>>
>> >>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>> >>> candygram.for.mongo@gmail.com> wrote:
>> >>>
>> >>> The ramBufferSizeMB was set to 6MB only on the test system to make the
>> >>> system crash sooner. In production that tag is commented out which
>> >>> I believe forces the default value to be used.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>> wrote:
>> >>>
>> >>> Hi,
>> >>> >
>> >>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>> >>> >
>> >>> >Ahmet
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>> >>> candygram.for.mongo@gmail.com> wrote:
>> >>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>> Exception
>> >>> >
>> >>> >*SOLR/Lucene version: *4.2.1*
>> >>> >
>> >>> >
>> >>> >*JVM version:
>> >>> >
>> >>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>> >>> >
>> >>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>> >>> >
>> >>> >
>> >>> >
>> >>> >*Indexer startup command:
>> >>> >
>> >>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>> >>> >
>> >>> >
>> >>> >
>> >>> >java " %JVMARGS% ^
>> >>> >
>> >>> >-Dcom.sun.management.jmxremote.port=1092 ^
>> >>> >
>> >>> >-Dcom.sun.management.jmxremote.ssl=false ^
>> >>> >
>> >>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>> >>> >
>> >>> >-jar start.jar
>> >>> >
>> >>> >
>> >>> >
>> >>> >*SOLR indexing HTTP parameters request:
>> >>> >
>> >>> >webapp=/solr path=/dataimport
>> >>> >params={clean=false&command=full-import&wt=javabin&version=2}
>> >>> >
>> >>> >
>> >>> >
>> >>> >We are getting a Java heap OOM exception when indexing (updating) 27
>> >>> >million records. If we increase the Java heap memory settings the
>> >>> problem
>> >>> >goes away but we believe the problem has not been fixed and that we
>> will
>> >>> >eventually get the same OOM exception. We have other processes on
>> the
>> >>> >server that also require resources so we cannot continually increase
>> the
>> >>> >memory settings to resolve the OOM issue. We are trying to find a
>> way to
>> >>> >configure the SOLR instance to reduce or preferably eliminate the
>> >>> >possibility of an OOM exception.
>> >>> >
>> >>> >
>> >>> >
>> >>> >We can reproduce the problem on a test machine. We set the Java heap
>> >>> >memory size to 64MB to accelerate the exception. If we increase this
>> >>> >setting the same problems occurs, just hours later. In the test
>> >>> >environment, we are using the following parameters:
>> >>> >
>> >>> >
>> >>> >
>> >>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> >>> >
>> >>> >
>> >>> >
>> >>> >Normally we use the default solrconfig.xml file with only the
>> following
>> >>> jar
>> >>> >file references added:
>> >>> >
>> >>> >
>> >>> >
>> >>> ><lib path="../../../../default/lib/common.jar" />
>> >>> >
>> >>> ><lib path="../../../../default/lib/webapp.jar" />
>> >>> >
>> >>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>> >>> >
>> >>> >
>> >>> >
>> >>> >Using these values and trying to index 6 million records from the
>> >>> database,
>> >>> >the Java Heap Out of Memory exception is thrown very quickly.
>> >>> >
>> >>> >
>> >>> >
>> >>> >We were able to complete a successful indexing by further modifying
>> the
>> >>> >solrconfig.xml and removing all or all but one <copyfield> tags from
>> the
>> >>> >schema.xml file.
>> >>> >
>> >>> >
>> >>> >
>> >>> >The following solrconfig.xml values were modified:
>> >>> >
>> >>> >
>> >>> >
>> >>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>> >>> >
>> >>> >
>> >>> >
>> >>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> >>> >
>> >>> ><int name="maxMergeAtOnce">2</int>
>> >>> >
>> >>> ><int name="maxMergeAtOnceExplicit">2</int>
>> >>> >
>> >>> ><int name="segmentsPerTier">10</int>
>> >>> >
>> >>> ><int name="maxMergedSegmentMB">150</int>
>> >>> >
>> >>> ></mergePolicy>
>> >>> >
>> >>> >
>> >>> >
>> >>> ><autoCommit>
>> >>> >
>> >>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
>> -- >
>> >>> >
>> >>> ><openSearcher>false</openSearcher>
>> >>> >
>> >>> ></autoCommit>
>> >>> >
>> >>> >
>> >>> >
>> >>> >Using our customized schema.xml file with two or more <copyfield>
>> tags,
>> >>> the
>> >>> >OOM exception is always thrown. Based on the errors, the problem
>> occurs
>> >>> >when the process was trying to do the merge. The error is provided
>> >>> below:
>> >>> >
>> >>> >
>> >>> >
>> >>> >Exception in thread "Lucene Merge Thread #156"
>> >>> >org.apache.lucene.index.MergePolicy$MergeException:
>> >>> >java.lang.OutOfMemoryError: Java heap space
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> >>> >
>> >>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>> >>> >
>> >>> > at
>> >>>
>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>> >>> >
>> >>> > at
>> >>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>> >>> >
>> >>> > at
>> >>>
>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>> >>> >
>> >>> > at
>> >>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> >>> >
>> >>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>> >>> >
>> >>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
>> writer
>> >>> >hit an OutOfMemoryError; cannot commit
>> >>> >
>> >>> > at
>> >>>
>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>> >>> >
>> >>> > at
>> >>>
>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>> >>> >
>> >>> > at
>> >>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>> >>> >
>> >>> > at
>> >>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>> >>> >
>> >>> > at
>> >>>
>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>> >
>> >>> > at
>> >>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>> >
>> >>> > at
>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >>> >
>> >>> > at
>> >>>
>> >>>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >>> >
>> >>> > at java.lang.Thread.run(Thread.java:722)
>> >>> >
>> >>> >
>> >>> >
>> >>> >We think but are not 100% sure that the problem is related to the
>> merge.
>> >>> >
>> >>> >
>> >>> >
>> >>> >Normally our schema.xml contains a lot of field specifications (like
>> the
>> >>> >ones seen in the file fragment below):
>> >>> >
>> >>> >
>> >>> >
>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >>> dest="ADDRESS.RECORD_ID.case_abc"
>> >>> >/>
>> >>> >
>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>> >>> >
>> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>> >>> >
>> >>> >
>> >>> >
>> >>> >In tests using the default file schema.xml and no <copyfield> tags,
>> >>> >indexing completed successfully. 6 million records produced a 900 MB
>> >>> data
>> >>> >directory.
>> >>> >
>> >>> >
>> >>> >
>> >>> >When I included just one <copyfield> tag, indexing completed
>> >>> successfully. 6
>> >>> >million records produced a 990 MB data directory (90 MB bigger).
>> >>> >
>> >>> >
>> >>> >
>> >>> >When I included just two <copyfield> tags, the index crashed with an
>> OOM
>> >>> >exception.
>> >>> >
>> >>> >
>> >>> >
>> >>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
>> postponed
>> >>> the
>> >>> >crash.
>> >>> >
>> >>> >
>> >>> >
>> >>> >The net of our test results I as follows:
>> >>> >
>> >>> >
>> >>> >
>> >>> >*solrconfig.xml*
>> >>> >
>> >>> >*schema.xml*
>> >>> >
>> >>> >*result*
>> >>> >
>> >>> >
>> >>> >default plus only jar references
>> >>> >
>> >>> >default (no copyfield tags)
>> >>> >
>> >>> >success
>> >>> >
>> >>> >default plus only jar references
>> >>> >
>> >>> >modified with one copyfield tag
>> >>> >
>> >>> >success
>> >>> >
>> >>> >default plus only jar references
>> >>> >
>> >>> >modified with two copyfield tags
>> >>> >
>> >>> >crash
>> >>> >
>> >>> >additional modified settings
>> >>> >
>> >>> >default (no copyfield tags)
>> >>> >
>> >>> >success
>> >>> >
>> >>> >additional modified settings
>> >>> >
>> >>> >modified with one copyfield tag
>> >>> >
>> >>> >success
>> >>> >
>> >>> >additional modified settings
>> >>> >
>> >>> >modified with two copyfield tags
>> >>> >
>> >>> >crash
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >Our question is, what can we do to eliminate these OOM exceptions?
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>> >
>>
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
I had similar problems before. We were trying to do same thing as you, fetching too many small documents from Oracle with dih. We were getting
Caused by: java.sql.SQLException: ORA-01652: unable to extend temp segment by 128 in tablespace TS_TEMP ORA-06512: at "IZCI.GET_FEED_KEYWORDS", line 20 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:459) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:193) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:197) at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1348) at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:635) at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:514) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:334) ... 12 more
DB admins explained it but I don't remember. At the end we sliced our SQL sentence and did smaller imports with clean=false.
Ahmet
On Monday, April 7, 2014 11:00 PM, Candygram For Mongo <> wrote:
I wanted to take a moment and say thank you for your help. We haven't
solved the problem yet but it seems like we may be on the path.
Responses to your questions below:
1) We are using settings of 6GBs for -Xmx and -Xms on a production server
where this process is failing on about 30 million relatively small records.
We have the need to execute the same processes on much larger data sets
(10x or more). There seems to be a somewhat linear requirement for memory
which is not sustainable.
2) We do not use the MDSolrDIHTransformer.jar. That jar is some legacy
code that is commented out. We are using the following jars:
common.jar, webapp.jar, commons-pool-1.4.jar.
The first two have our custom code in it that include filters. The last
is from Apache.
3) We have Solr configured to switch what it uses based on the environment.
Looking at the INFOSTREAM.txt file, it is using MMap in the environment in
question.
4) Incrementing the batchSize to 5,000 or 10,000 accelerates the OOM error
(using the 64MB heap size) and it is not able to execute the query. See
the error below:
*java.sql.SQLException: Protocol violation: [2]*
* at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:527)*
* at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)*
* at
oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)*
* at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:574)*
* at
oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:4011)*
* at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:410)*
* at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:395)*
* at
org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:284)*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)*
* at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)*
* at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)*
* at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)*
*Apr 07, 2014 11:11:54 AM org.apache.solr.common.SolrException log*
*SEVERE: Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.solr.handler.dataimport.Data*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:266)*
* at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)*
* at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)*
* at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)*
*Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemor*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:406)*
* at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)*
* ... 3 more*
*Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)*
We also suspect that the copyfield may be the culprit. We are trying the
CSV process now.
On Sat, Apr 5, 2014 at 3:16 AM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> Now we have a more informative error
> : org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
>
> 1) Does this happen when you increase -Xmx64m -Xms64m ?
>
> 2) I see you use custom jars called "MDSolrDIHTransformer JARs inside"
> But I don't see any Transformers used in database.xm, why is that. I would
> remove them just to be sure.
>
> 3) I see you have org.apache.solr.core.StandardDirectoryFactory declared
> in sorlconfig. Assuming you are using, 64 bit windows, it is recommended to
> use MMap
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
>
> 4) In your previous mail you had batch size set, now there is not
> batchSize defined in database.xml. For MySQL it is recommended to use -1.
> Not sure about oracle, I personally used 10,000 once for Oracle.
> http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F
>
> You have a lot of copyFields defined. There could be some gotchas when
> handling unusually much copy fields. I would really try CSV option here.
> Given that you have only full import SQL defined and it is not a complex
> one. It queries only one table. I believe Oracle has some tool to export a
> table to CSV file efficiently.
>
> On Saturday, April 5, 2014 3:05 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
> Does this user list allow attachments? I have four files attached
> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
> process again using the parameters you suggested, but not to a csv file.
> It errored out quickly. We are working on the csv file run.
>
> Removed both <autoCommit> and
> <autoSoftCommit> parts/definitions from solrconfig.xml
>
> Disabled tlog by removing
> <updateLog>
> <str
> name="dir">${solr.ulog.dir:}</str>
> </updateLog>
>
> from solrconfig.xml
>
> Used commit=true parameter.
> ?commit=true&command=full-import
>
>
>
>
> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
> Hi,
> >
> >This may not solve your problem but generally it is recommended to
> disable auto commit and transaction logs for bulk indexing.
> >And issue one commit at the very end. Do you tlogs enabled? I see "commit
> failed" in the error message thats why I am offering this.
> >
> >And regarding comma separated values, with this approach you focus on
> just solr importing process. You separate data acquisition phrase. And it
> is very fast load even big csv files
> http://wiki.apache.org/solr/UpdateCSV
> >I have never experienced OOM during indexing, I suspect data acquisition
> has role in it.
> >
> >Ahmet
> >
> >
> >On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >
> >We would be happy to try that. That sounds counter intuitive for the
> high volume of records we have. Can you help me understand how that might
> solve our problem?
> >
> >
> >
> >
> >On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> >
> >Hi,
> >>
> >>Can you remove auto commit for bulk import. Commit at the very end?
> >>
> >>Ahmet
> >>
> >>
> >>
> >>
> >>On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >>In case the attached database.xml file didn't show up, I have pasted in
> the
> >>contents below:
> >>
> >><dataConfig>
> >><dataSource
> >>name="org_only"
> >>type="JdbcDataSource"
> >>driver="oracle.jdbc.OracleDriver"
> >>url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
> >>user="admin"
> >>password="admin"
> >>readOnly="false"
> >>batchSize="100"
> >>/>
> >><document>
> >>
> >>
> >><entity name="full-index" query="
> >>select
> >>
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
> >>as SOLR_ID,
> >>
> >>'ORCL.ADDRESS_ACCT_ALL'
> >>as SOLR_CATEGORY,
> >>
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
> >>ADDRESSALLROWID,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> >>ADDRESSALLADDRTYPECD,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
> >>ADDRESSALLLONGITUDE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
> >>ADDRESSALLLATITUDE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
> >>ADDRESSALLADDRNAME,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
> >>ADDRESSALLCITY,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
> >>ADDRESSALLSTATE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> >>ADDRESSALLEMAILADDR
> >>
> >>from ORCL.ADDRESS_ACCT_ALL
> >>" >
> >>
> >><field column="SOLR_ID" name="id" />
> >><field column="SOLR_CATEGORY" name="category" />
> >><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
> >><field column="ADDRESSALLADDRTYPECD"
> >>name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
> >><field column="ADDRESSALLLONGITUDE"
> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
> >><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc"
> />
> >><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
> />
> >><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
> >><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
> >><field column="ADDRESSALLEMAILADDR"
> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
> >>/>
> >>
> >></entity>
> >>
> >>
> >>
> >><!-- Varaibles -->
> >><!-- '${dataimporter.last_index_time}' -->
> >></document>
> >></dataConfig>
> >>
> >>
> >>
> >>
> >>
> >>
> >>On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
> >>candygram.for.mongo@gmail.com> wrote:
> >>
> >>> In this case we are indexing an Oracle database.
> >>>
> >>> We do not include the data-config.xml in our distribution. We store
> the
> >>> database information in the database.xml file. I have attached the
> >>> database.xml file.
> >>>
> >>> When we use the default merge policy settings, we get the same results.
> >>>
> >>>
> >>>
> >>> We have not tried to dump the table to a comma separated file. We
> think
> >>> that dumping this size table to disk will introduce other memory
> problems
> >>> with big file management. We have not tested that case.
> >>>
> >>>
> >>> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Which database are you using? Can you send us data-config.xml?
> >>>>
> >>>> What happens when you use default merge policy settings?
> >>>>
> >>>> What happens when you dump your table to Comma Separated File and fed
> >>>> that file to solr?
> >>>>
> >>>> Ahmet
> >>>>
> >>>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> >>>> candygram.for.mongo@gmail.com> wrote:
> >>>>
> >>>> The ramBufferSizeMB was set to 6MB only on the test system to make the
> >>>> system crash sooner. In production that tag is commented out which
> >>>> I believe forces the default value to be used.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>> >
> >>>> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >>>> >
> >>>> >Ahmet
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> >>>> candygram.for.mongo@gmail.com> wrote:
> >>>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
> Exception
> >>>> >
> >>>> >*SOLR/Lucene version: *4.2.1*
> >>>> >
> >>>> >
> >>>> >*JVM version:
> >>>> >
> >>>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >>>> >
> >>>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >>>> >
> >>>> >
> >>>> >
> >>>> >*Indexer startup command:
> >>>> >
> >>>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >>>> >
> >>>> >
> >>>> >
> >>>> >java " %JVMARGS% ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.port=1092 ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.ssl=false ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >>>> >
> >>>> >-jar start.jar
> >>>> >
> >>>> >
> >>>> >
> >>>> >*SOLR indexing HTTP parameters request:
> >>>> >
> >>>> >webapp=/solr path=/dataimport
> >>>> >params={clean=false&command=full-import&wt=javabin&version=2}
> >>>> >
> >>>> >
> >>>> >
> >>>> >We are getting a Java heap OOM exception when indexing (updating) 27
> >>>> >million records. If we increase the Java heap memory settings the
> >>>> problem
> >>>> >goes away but we believe the problem has not been fixed and that we
> will
> >>>> >eventually get the same OOM exception. We have other processes on
> the
> >>>> >server that also require resources so we cannot continually increase
> the
> >>>> >memory settings to resolve the OOM issue. We are trying to find a
> way to
> >>>> >configure the SOLR instance to reduce or preferably eliminate the
> >>>> >possibility of an OOM exception.
> >>>> >
> >>>> >
> >>>> >
> >>>> >We can reproduce the problem on a test machine. We set the Java heap
> >>>> >memory size to 64MB to accelerate the exception. If we increase this
> >>>> >setting the same problems occurs, just hours later. In the test
> >>>> >environment, we are using the following parameters:
> >>>> >
> >>>> >
> >>>> >
> >>>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >>>> >
> >>>> >
> >>>> >
> >>>> >Normally we use the default solrconfig.xml file with only the
> following
> >>>> jar
> >>>> >file references added:
> >>>> >
> >>>> >
> >>>> >
> >>>> ><lib path="../../../../default/lib/common.jar" />
> >>>> >
> >>>> ><lib path="../../../../default/lib/webapp.jar" />
> >>>> >
> >>>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >>>> >
> >>>> >
> >>>> >
> >>>> >Using these values and trying to index 6 million records from the
> >>>> database,
> >>>> >the Java Heap Out of Memory exception is thrown very quickly.
> >>>> >
> >>>> >
> >>>> >
> >>>> >We were able to complete a successful indexing by further modifying
> the
> >>>> >solrconfig.xml and removing all or all but one <copyfield> tags from
> the
> >>>> >schema.xml file.
> >>>> >
> >>>> >
> >>>> >
> >>>> >The following solrconfig.xml values were modified:
> >>>> >
> >>>> >
> >>>> >
> >>>> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >>>> >
> >>>> >
> >>>> >
> >>>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >>>> >
> >>>> ><int name="maxMergeAtOnce">2</int>
> >>>> >
> >>>> ><int name="maxMergeAtOnceExplicit">2</int>
> >>>> >
> >>>> ><int name="segmentsPerTier">10</int>
> >>>> >
> >>>> ><int name="maxMergedSegmentMB">150</int>
> >>>> >
> >>>> ></mergePolicy>
> >>>> >
> >>>> >
> >>>> >
> >>>> ><autoCommit>
> >>>> >
> >>>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
> -- >
> >>>> >
> >>>> ><openSearcher>false</openSearcher>
> >>>> >
> >>>> ></autoCommit>
> >>>> >
> >>>> >
> >>>> >
> >>>> >Using our customized schema.xml file with two or more <copyfield>
> tags,
> >>>> the
> >>>> >OOM exception is always thrown. Based on the errors, the problem
> occurs
> >>>> >when the process was trying to do the merge. The error is provided
> >>>> below:
> >>>> >
> >>>> >
> >>>> >
> >>>> >Exception in thread "Lucene Merge Thread #156"
> >>>> >org.apache.lucene.index.MergePolicy$MergeException:
> >>>> >java.lang.OutOfMemoryError: Java heap space
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >>>> >
> >>>> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >>>> >
> >>>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >>>> >
> >>>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
> writer
> >>>> >hit an OutOfMemoryError; cannot commit
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >>>> >
> >>>> > at
> >>>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >>>> >
> >>>> > at
> >>>>
> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> >
> >>>> > at
> >>>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >>>> >
> >>>> > at
> >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>>> >
> >>>> > at java.lang.Thread.run(Thread.java:722)
> >>>> >
> >>>> >
> >>>> >
> >>>> >We think but are not 100% sure that the problem is related to the
> merge.
> >>>> >
> >>>> >
> >>>> >
> >>>> >Normally our schema.xml contains a lot of field specifications (like
> the
> >>>> >ones seen in the file fragment below):
> >>>> >
> >>>> >
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> dest="ADDRESS.RECORD_ID.case_abc"
> >>>> >/>
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >>>> >
> >>>> >
> >>>> >
> >>>> >In tests using the default file schema.xml and no <copyfield> tags,
> >>>> >indexing completed successfully. 6 million records produced a 900 MB
> >>>> data
> >>>> >directory.
> >>>> >
> >>>> >
> >>>> >
> >>>> >When I included just one <copyfield> tag, indexing completed
> >>>> successfully. 6
> >>>> >million records produced a 990 MB data directory (90 MB bigger).
> >>>> >
> >>>> >
> >>>> >
> >>>> >When I included just two <copyfield> tags, the index crashed with an
> OOM
> >>>> >exception.
> >>>> >
> >>>> >
> >>>> >
> >>>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
> postponed
> >>>> the
> >>>> >crash.
> >>>> >
> >>>> >
> >>>> >
> >>>> >The net of our test results I as follows:
> >>>> >
> >>>> >
> >>>> >
> >>>> >*solrconfig.xml*
> >>>> >
> >>>> >*schema.xml*
> >>>> >
> >>>> >*result*
> >>>> >
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >default (no copyfield tags)
> >>>> >
> >>>> >success
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >modified with one copyfield tag
> >>>> >
> >>>> >success
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >modified with two copyfield tags
> >>>> >
> >>>> >crash
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >default (no copyfield tags)
> >>>> >
> >>>> >success
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >modified with one copyfield tag
> >>>> >
> >>>> >success
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >modified with two copyfield tags
> >>>> >
> >>>> >crash
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >Our question is, what can we do to eliminate these OOM exceptions?
> >>>> >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >>
> >
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
I wanted to take a moment and say thank you for your help. We haven't
solved the problem yet but it seems like we may be on the path.
Responses to your questions below:
1) We are using settings of 6GBs for -Xmx and -Xms on a production server
where this process is failing on about 30 million relatively small records.
We have the need to execute the same processes on much larger data sets
(10x or more). There seems to be a somewhat linear requirement for memory
which is not sustainable.
2) We do not use the MDSolrDIHTransformer.jar. That jar is some legacy
code that is commented out. We are using the following jars:
common.jar, webapp.jar, commons-pool-1.4.jar.
The first two have our custom code in it that include filters. The last
is from Apache.
3) We have Solr configured to switch what it uses based on the environment.
Looking at the INFOSTREAM.txt file, it is using MMap in the environment in
question.
4) Incrementing the batchSize to 5,000 or 10,000 accelerates the OOM error
(using the 64MB heap size) and it is not able to execute the query. See
the error below:
*java.sql.SQLException: Protocol violation: [2]*
* at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:527)*
* at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)*
* at
oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)*
* at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:574)*
* at
oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:4011)*
* at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:410)*
* at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:395)*
* at
org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:284)*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)*
* at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)*
* at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)*
* at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)*
*Apr 07, 2014 11:11:54 AM org.apache.solr.common.SolrException log*
*SEVERE: Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.solr.handler.dataimport.Data*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:266)*
* at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:422)*
* at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)*
* at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)*
*Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemor*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:406)*
* at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:319)*
* at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:227)*
* ... 3 more*
*Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)*
* at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)*
We also suspect that the copyfield may be the culprit. We are trying the
CSV process now.
On Sat, Apr 5, 2014 at 3:16 AM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> Now we have a more informative error
> : org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
>
> 1) Does this happen when you increase -Xmx64m -Xms64m ?
>
> 2) I see you use custom jars called "MDSolrDIHTransformer JARs inside"
> But I don't see any Transformers used in database.xm, why is that. I would
> remove them just to be sure.
>
> 3) I see you have org.apache.solr.core.StandardDirectoryFactory declared
> in sorlconfig. Assuming you are using, 64 bit windows, it is recommended to
> use MMap
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
>
> 4) In your previous mail you had batch size set, now there is not
> batchSize defined in database.xml. For MySQL it is recommended to use -1.
> Not sure about oracle, I personally used 10,000 once for Oracle.
> http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F
>
> You have a lot of copyFields defined. There could be some gotchas when
> handling unusually much copy fields. I would really try CSV option here.
> Given that you have only full import SQL defined and it is not a complex
> one. It queries only one table. I believe Oracle has some tool to export a
> table to CSV file efficiently.
>
> On Saturday, April 5, 2014 3:05 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
> Does this user list allow attachments? I have four files attached
> (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
> process again using the parameters you suggested, but not to a csv file.
> It errored out quickly. We are working on the csv file run.
>
> Removed both <autoCommit> and
> <autoSoftCommit> parts/definitions from solrconfig.xml
>
> Disabled tlog by removing
> <updateLog>
> <str
> name="dir">${solr.ulog.dir:}</str>
> </updateLog>
>
> from solrconfig.xml
>
> Used commit=true parameter.
> ?commit=true&command=full-import
>
>
>
>
> On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
> Hi,
> >
> >This may not solve your problem but generally it is recommended to
> disable auto commit and transaction logs for bulk indexing.
> >And issue one commit at the very end. Do you tlogs enabled? I see "commit
> failed" in the error message thats why I am offering this.
> >
> >And regarding comma separated values, with this approach you focus on
> just solr importing process. You separate data acquisition phrase. And it
> is very fast load even big csv files
> http://wiki.apache.org/solr/UpdateCSV
> >I have never experienced OOM during indexing, I suspect data acquisition
> has role in it.
> >
> >Ahmet
> >
> >
> >On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >
> >We would be happy to try that. That sounds counter intuitive for the
> high volume of records we have. Can you help me understand how that might
> solve our problem?
> >
> >
> >
> >
> >On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> >
> >Hi,
> >>
> >>Can you remove auto commit for bulk import. Commit at the very end?
> >>
> >>Ahmet
> >>
> >>
> >>
> >>
> >>On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >>In case the attached database.xml file didn't show up, I have pasted in
> the
> >>contents below:
> >>
> >><dataConfig>
> >><dataSource
> >>name="org_only"
> >>type="JdbcDataSource"
> >>driver="oracle.jdbc.OracleDriver"
> >>url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
> >>user="admin"
> >>password="admin"
> >>readOnly="false"
> >>batchSize="100"
> >>/>
> >><document>
> >>
> >>
> >><entity name="full-index" query="
> >>select
> >>
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
> >>as SOLR_ID,
> >>
> >>'ORCL.ADDRESS_ACCT_ALL'
> >>as SOLR_CATEGORY,
> >>
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
> >>ADDRESSALLROWID,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> >>ADDRESSALLADDRTYPECD,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
> >>ADDRESSALLLONGITUDE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
> >>ADDRESSALLLATITUDE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
> >>ADDRESSALLADDRNAME,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
> >>ADDRESSALLCITY,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
> >>ADDRESSALLSTATE,
> >>NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> >>ADDRESSALLEMAILADDR
> >>
> >>from ORCL.ADDRESS_ACCT_ALL
> >>" >
> >>
> >><field column="SOLR_ID" name="id" />
> >><field column="SOLR_CATEGORY" name="category" />
> >><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
> >><field column="ADDRESSALLADDRTYPECD"
> >>name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
> >><field column="ADDRESSALLLONGITUDE"
> name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
> >><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc"
> />
> >><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
> />
> >><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
> >><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
> >><field column="ADDRESSALLEMAILADDR"
> name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
> >>/>
> >>
> >></entity>
> >>
> >>
> >>
> >><!-- Varaibles -->
> >><!-- '${dataimporter.last_index_time}' -->
> >></document>
> >></dataConfig>
> >>
> >>
> >>
> >>
> >>
> >>
> >>On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
> >>candygram.for.mongo@gmail.com> wrote:
> >>
> >>> In this case we are indexing an Oracle database.
> >>>
> >>> We do not include the data-config.xml in our distribution. We store
> the
> >>> database information in the database.xml file. I have attached the
> >>> database.xml file.
> >>>
> >>> When we use the default merge policy settings, we get the same results.
> >>>
> >>>
> >>>
> >>> We have not tried to dump the table to a comma separated file. We
> think
> >>> that dumping this size table to disk will introduce other memory
> problems
> >>> with big file management. We have not tested that case.
> >>>
> >>>
> >>> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Which database are you using? Can you send us data-config.xml?
> >>>>
> >>>> What happens when you use default merge policy settings?
> >>>>
> >>>> What happens when you dump your table to Comma Separated File and fed
> >>>> that file to solr?
> >>>>
> >>>> Ahmet
> >>>>
> >>>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> >>>> candygram.for.mongo@gmail.com> wrote:
> >>>>
> >>>> The ramBufferSizeMB was set to 6MB only on the test system to make the
> >>>> system crash sooner. In production that tag is commented out which
> >>>> I believe forces the default value to be used.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>> >
> >>>> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >>>> >
> >>>> >Ahmet
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> >>>> candygram.for.mongo@gmail.com> wrote:
> >>>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
> Exception
> >>>> >
> >>>> >*SOLR/Lucene version: *4.2.1*
> >>>> >
> >>>> >
> >>>> >*JVM version:
> >>>> >
> >>>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >>>> >
> >>>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >>>> >
> >>>> >
> >>>> >
> >>>> >*Indexer startup command:
> >>>> >
> >>>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >>>> >
> >>>> >
> >>>> >
> >>>> >java " %JVMARGS% ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.port=1092 ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.ssl=false ^
> >>>> >
> >>>> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >>>> >
> >>>> >-jar start.jar
> >>>> >
> >>>> >
> >>>> >
> >>>> >*SOLR indexing HTTP parameters request:
> >>>> >
> >>>> >webapp=/solr path=/dataimport
> >>>> >params={clean=false&command=full-import&wt=javabin&version=2}
> >>>> >
> >>>> >
> >>>> >
> >>>> >We are getting a Java heap OOM exception when indexing (updating) 27
> >>>> >million records. If we increase the Java heap memory settings the
> >>>> problem
> >>>> >goes away but we believe the problem has not been fixed and that we
> will
> >>>> >eventually get the same OOM exception. We have other processes on
> the
> >>>> >server that also require resources so we cannot continually increase
> the
> >>>> >memory settings to resolve the OOM issue. We are trying to find a
> way to
> >>>> >configure the SOLR instance to reduce or preferably eliminate the
> >>>> >possibility of an OOM exception.
> >>>> >
> >>>> >
> >>>> >
> >>>> >We can reproduce the problem on a test machine. We set the Java heap
> >>>> >memory size to 64MB to accelerate the exception. If we increase this
> >>>> >setting the same problems occurs, just hours later. In the test
> >>>> >environment, we are using the following parameters:
> >>>> >
> >>>> >
> >>>> >
> >>>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >>>> >
> >>>> >
> >>>> >
> >>>> >Normally we use the default solrconfig.xml file with only the
> following
> >>>> jar
> >>>> >file references added:
> >>>> >
> >>>> >
> >>>> >
> >>>> ><lib path="../../../../default/lib/common.jar" />
> >>>> >
> >>>> ><lib path="../../../../default/lib/webapp.jar" />
> >>>> >
> >>>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >>>> >
> >>>> >
> >>>> >
> >>>> >Using these values and trying to index 6 million records from the
> >>>> database,
> >>>> >the Java Heap Out of Memory exception is thrown very quickly.
> >>>> >
> >>>> >
> >>>> >
> >>>> >We were able to complete a successful indexing by further modifying
> the
> >>>> >solrconfig.xml and removing all or all but one <copyfield> tags from
> the
> >>>> >schema.xml file.
> >>>> >
> >>>> >
> >>>> >
> >>>> >The following solrconfig.xml values were modified:
> >>>> >
> >>>> >
> >>>> >
> >>>> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >>>> >
> >>>> >
> >>>> >
> >>>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >>>> >
> >>>> ><int name="maxMergeAtOnce">2</int>
> >>>> >
> >>>> ><int name="maxMergeAtOnceExplicit">2</int>
> >>>> >
> >>>> ><int name="segmentsPerTier">10</int>
> >>>> >
> >>>> ><int name="maxMergedSegmentMB">150</int>
> >>>> >
> >>>> ></mergePolicy>
> >>>> >
> >>>> >
> >>>> >
> >>>> ><autoCommit>
> >>>> >
> >>>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
> -- >
> >>>> >
> >>>> ><openSearcher>false</openSearcher>
> >>>> >
> >>>> ></autoCommit>
> >>>> >
> >>>> >
> >>>> >
> >>>> >Using our customized schema.xml file with two or more <copyfield>
> tags,
> >>>> the
> >>>> >OOM exception is always thrown. Based on the errors, the problem
> occurs
> >>>> >when the process was trying to do the merge. The error is provided
> >>>> below:
> >>>> >
> >>>> >
> >>>> >
> >>>> >Exception in thread "Lucene Merge Thread #156"
> >>>> >org.apache.lucene.index.MergePolicy$MergeException:
> >>>> >java.lang.OutOfMemoryError: Java heap space
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >>>> >
> >>>> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >>>> >
> >>>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >>>> >
> >>>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
> writer
> >>>> >hit an OutOfMemoryError; cannot commit
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >>>> >
> >>>> > at
> >>>>
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >>>> >
> >>>> > at
> >>>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >>>> >
> >>>> > at
> >>>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >>>> >
> >>>> > at
> >>>>
> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> >
> >>>> > at
> >>>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >>>> >
> >>>> > at
> >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >>>> >
> >>>> > at
> >>>>
> >>>>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>>> >
> >>>> > at java.lang.Thread.run(Thread.java:722)
> >>>> >
> >>>> >
> >>>> >
> >>>> >We think but are not 100% sure that the problem is related to the
> merge.
> >>>> >
> >>>> >
> >>>> >
> >>>> >Normally our schema.xml contains a lot of field specifications (like
> the
> >>>> >ones seen in the file fragment below):
> >>>> >
> >>>> >
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> dest="ADDRESS.RECORD_ID.case_abc"
> >>>> >/>
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >>>> >
> >>>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >>>> >
> >>>> >
> >>>> >
> >>>> >In tests using the default file schema.xml and no <copyfield> tags,
> >>>> >indexing completed successfully. 6 million records produced a 900 MB
> >>>> data
> >>>> >directory.
> >>>> >
> >>>> >
> >>>> >
> >>>> >When I included just one <copyfield> tag, indexing completed
> >>>> successfully. 6
> >>>> >million records produced a 990 MB data directory (90 MB bigger).
> >>>> >
> >>>> >
> >>>> >
> >>>> >When I included just two <copyfield> tags, the index crashed with an
> OOM
> >>>> >exception.
> >>>> >
> >>>> >
> >>>> >
> >>>> >Changing parameters like maxMergedSegmentMB or maxDocs, only
> postponed
> >>>> the
> >>>> >crash.
> >>>> >
> >>>> >
> >>>> >
> >>>> >The net of our test results I as follows:
> >>>> >
> >>>> >
> >>>> >
> >>>> >*solrconfig.xml*
> >>>> >
> >>>> >*schema.xml*
> >>>> >
> >>>> >*result*
> >>>> >
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >default (no copyfield tags)
> >>>> >
> >>>> >success
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >modified with one copyfield tag
> >>>> >
> >>>> >success
> >>>> >
> >>>> >default plus only jar references
> >>>> >
> >>>> >modified with two copyfield tags
> >>>> >
> >>>> >crash
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >default (no copyfield tags)
> >>>> >
> >>>> >success
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >modified with one copyfield tag
> >>>> >
> >>>> >success
> >>>> >
> >>>> >additional modified settings
> >>>> >
> >>>> >modified with two copyfield tags
> >>>> >
> >>>> >crash
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >Our question is, what can we do to eliminate these OOM exceptions?
> >>>> >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >>
> >
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
Now we have a more informative error : org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:404)
1) Does this happen when you increase -Xmx64m -Xms64m ?
2) I see you use custom jars called "MDSolrDIHTransformer JARs inside" But I don't see any Transformers used in database.xm, why is that. I would remove them just to be sure.
3) I see you have org.apache.solr.core.StandardDirectoryFactory declared in sorlconfig. Assuming you are using, 64 bit windows, it is recommended to use MMap
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
4) In your previous mail you had batch size set, now there is not batchSize defined in database.xml. For MySQL it is recommended to use -1. Not sure about oracle, I personally used 10,000 once for Oracle. http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F
You have a lot of copyFields defined. There could be some gotchas when handling unusually much copy fields. I would really try CSV option here. Given that you have only full import SQL defined and it is not a complex one. It queries only one table. I believe Oracle has some tool to export a table to CSV file efficiently.
On Saturday, April 5, 2014 3:05 AM, Candygram For Mongo <ca...@gmail.com> wrote:
Does this user list allow attachments? I have four files attached (database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the process again using the parameters you suggested, but not to a csv file. It errored out quickly. We are working on the csv file run.
Removed both <autoCommit> and
<autoSoftCommit> parts/definitions from solrconfig.xml
Disabled tlog by removing
<updateLog>
<str
name="dir">${solr.ulog.dir:}</str>
</updateLog>
from solrconfig.xml
Used commit=true parameter.
?commit=true&command=full-import
On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
Hi,
>
>This may not solve your problem but generally it is recommended to disable auto commit and transaction logs for bulk indexing.
>And issue one commit at the very end. Do you tlogs enabled? I see "commit failed" in the error message thats why I am offering this.
>
>And regarding comma separated values, with this approach you focus on just solr importing process. You separate data acquisition phrase. And it is very fast load even big csv files http://wiki.apache.org/solr/UpdateCSV
>I have never experienced OOM during indexing, I suspect data acquisition has role in it.
>
>Ahmet
>
>
>On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <ca...@gmail.com> wrote:
>
>We would be happy to try that. That sounds counter intuitive for the high volume of records we have. Can you help me understand how that might solve our problem?
>
>
>
>
>On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>Hi,
>>
>>Can you remove auto commit for bulk import. Commit at the very end?
>>
>>Ahmet
>>
>>
>>
>>
>>On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <ca...@gmail.com> wrote:
>>In case the attached database.xml file didn't show up, I have pasted in the
>>contents below:
>>
>><dataConfig>
>><dataSource
>>name="org_only"
>>type="JdbcDataSource"
>>driver="oracle.jdbc.OracleDriver"
>>url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>>user="admin"
>>password="admin"
>>readOnly="false"
>>batchSize="100"
>>/>
>><document>
>>
>>
>><entity name="full-index" query="
>>select
>>
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>>as SOLR_ID,
>>
>>'ORCL.ADDRESS_ACCT_ALL'
>>as SOLR_CATEGORY,
>>
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>>ADDRESSALLROWID,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>>ADDRESSALLADDRTYPECD,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>>ADDRESSALLLONGITUDE,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>>ADDRESSALLLATITUDE,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>>ADDRESSALLADDRNAME,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>>ADDRESSALLCITY,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>>ADDRESSALLSTATE,
>>NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>>ADDRESSALLEMAILADDR
>>
>>from ORCL.ADDRESS_ACCT_ALL
>>" >
>>
>><field column="SOLR_ID" name="id" />
>><field column="SOLR_CATEGORY" name="category" />
>><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
>><field column="ADDRESSALLADDRTYPECD"
>>name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>><field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
>><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
>><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
>><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>><field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>>/>
>>
>></entity>
>>
>>
>>
>><!-- Varaibles -->
>><!-- '${dataimporter.last_index_time}' -->
>></document>
>></dataConfig>
>>
>>
>>
>>
>>
>>
>>On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>>candygram.for.mongo@gmail.com> wrote:
>>
>>> In this case we are indexing an Oracle database.
>>>
>>> We do not include the data-config.xml in our distribution. We store the
>>> database information in the database.xml file. I have attached the
>>> database.xml file.
>>>
>>> When we use the default merge policy settings, we get the same results.
>>>
>>>
>>>
>>> We have not tried to dump the table to a comma separated file. We think
>>> that dumping this size table to disk will introduce other memory problems
>>> with big file management. We have not tested that case.
>>>
>>>
>>> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Which database are you using? Can you send us data-config.xml?
>>>>
>>>> What happens when you use default merge policy settings?
>>>>
>>>> What happens when you dump your table to Comma Separated File and fed
>>>> that file to solr?
>>>>
>>>> Ahmet
>>>>
>>>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>>>> candygram.for.mongo@gmail.com> wrote:
>>>>
>>>> The ramBufferSizeMB was set to 6MB only on the test system to make the
>>>> system crash sooner. In production that tag is commented out which
>>>> I believe forces the default value to be used.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>>
>>>> Hi,
>>>> >
>>>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>>>> >
>>>> >Ahmet
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>>>> candygram.for.mongo@gmail.com> wrote:
>>>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>>>> >
>>>> >*SOLR/Lucene version: *4.2.1*
>>>> >
>>>> >
>>>> >*JVM version:
>>>> >
>>>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>>>> >
>>>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>>>> >
>>>> >
>>>> >
>>>> >*Indexer startup command:
>>>> >
>>>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>>>> >
>>>> >
>>>> >
>>>> >java " %JVMARGS% ^
>>>> >
>>>> >-Dcom.sun.management.jmxremote.port=1092 ^
>>>> >
>>>> >-Dcom.sun.management.jmxremote.ssl=false ^
>>>> >
>>>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>>>> >
>>>> >-jar start.jar
>>>> >
>>>> >
>>>> >
>>>> >*SOLR indexing HTTP parameters request:
>>>> >
>>>> >webapp=/solr path=/dataimport
>>>> >params={clean=false&command=full-import&wt=javabin&version=2}
>>>> >
>>>> >
>>>> >
>>>> >We are getting a Java heap OOM exception when indexing (updating) 27
>>>> >million records. If we increase the Java heap memory settings the
>>>> problem
>>>> >goes away but we believe the problem has not been fixed and that we will
>>>> >eventually get the same OOM exception. We have other processes on the
>>>> >server that also require resources so we cannot continually increase the
>>>> >memory settings to resolve the OOM issue. We are trying to find a way to
>>>> >configure the SOLR instance to reduce or preferably eliminate the
>>>> >possibility of an OOM exception.
>>>> >
>>>> >
>>>> >
>>>> >We can reproduce the problem on a test machine. We set the Java heap
>>>> >memory size to 64MB to accelerate the exception. If we increase this
>>>> >setting the same problems occurs, just hours later. In the test
>>>> >environment, we are using the following parameters:
>>>> >
>>>> >
>>>> >
>>>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>>>> >
>>>> >
>>>> >
>>>> >Normally we use the default solrconfig.xml file with only the following
>>>> jar
>>>> >file references added:
>>>> >
>>>> >
>>>> >
>>>> ><lib path="../../../../default/lib/common.jar" />
>>>> >
>>>> ><lib path="../../../../default/lib/webapp.jar" />
>>>> >
>>>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>>>> >
>>>> >
>>>> >
>>>> >Using these values and trying to index 6 million records from the
>>>> database,
>>>> >the Java Heap Out of Memory exception is thrown very quickly.
>>>> >
>>>> >
>>>> >
>>>> >We were able to complete a successful indexing by further modifying the
>>>> >solrconfig.xml and removing all or all but one <copyfield> tags from the
>>>> >schema.xml file.
>>>> >
>>>> >
>>>> >
>>>> >The following solrconfig.xml values were modified:
>>>> >
>>>> >
>>>> >
>>>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>>>> >
>>>> >
>>>> >
>>>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>>> >
>>>> ><int name="maxMergeAtOnce">2</int>
>>>> >
>>>> ><int name="maxMergeAtOnceExplicit">2</int>
>>>> >
>>>> ><int name="segmentsPerTier">10</int>
>>>> >
>>>> ><int name="maxMergedSegmentMB">150</int>
>>>> >
>>>> ></mergePolicy>
>>>> >
>>>> >
>>>> >
>>>> ><autoCommit>
>>>> >
>>>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>>>> >
>>>> ><openSearcher>false</openSearcher>
>>>> >
>>>> ></autoCommit>
>>>> >
>>>> >
>>>> >
>>>> >Using our customized schema.xml file with two or more <copyfield> tags,
>>>> the
>>>> >OOM exception is always thrown. Based on the errors, the problem occurs
>>>> >when the process was trying to do the merge. The error is provided
>>>> below:
>>>> >
>>>> >
>>>> >
>>>> >Exception in thread "Lucene Merge Thread #156"
>>>> >org.apache.lucene.index.MergePolicy$MergeException:
>>>> >java.lang.OutOfMemoryError: Java heap space
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>>> >
>>>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>>> >
>>>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>>>> >
>>>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>>>> >hit an OutOfMemoryError; cannot commit
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>>>> >
>>>> > at
>>>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>>>> >
>>>> > at
>>>>
>>>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>>>> >
>>>> > at
>>>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>>>> >
>>>> > at
>>>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> >
>>>> > at
>>>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>> >
>>>> > at
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>> >
>>>> > at
>>>>
>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> >
>>>> > at
>>>>
>>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> >
>>>> > at
>>>>
>>>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> >
>>>> > at
>>>>
>>>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> >
>>>> > at java.lang.Thread.run(Thread.java:722)
>>>> >
>>>> >
>>>> >
>>>> >We think but are not 100% sure that the problem is related to the merge.
>>>> >
>>>> >
>>>> >
>>>> >Normally our schema.xml contains a lot of field specifications (like the
>>>> >ones seen in the file fragment below):
>>>> >
>>>> >
>>>> >
>>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> dest="ADDRESS.RECORD_ID.case_abc"
>>>> >/>
>>>> >
>>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>>>> >
>>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>>>> >
>>>> >
>>>> >
>>>> >In tests using the default file schema.xml and no <copyfield> tags,
>>>> >indexing completed successfully. 6 million records produced a 900 MB
>>>> data
>>>> >directory.
>>>> >
>>>> >
>>>> >
>>>> >When I included just one <copyfield> tag, indexing completed
>>>> successfully. 6
>>>> >million records produced a 990 MB data directory (90 MB bigger).
>>>> >
>>>> >
>>>> >
>>>> >When I included just two <copyfield> tags, the index crashed with an OOM
>>>> >exception.
>>>> >
>>>> >
>>>> >
>>>> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>>>> the
>>>> >crash.
>>>> >
>>>> >
>>>> >
>>>> >The net of our test results I as follows:
>>>> >
>>>> >
>>>> >
>>>> >*solrconfig.xml*
>>>> >
>>>> >*schema.xml*
>>>> >
>>>> >*result*
>>>> >
>>>> >
>>>> >default plus only jar references
>>>> >
>>>> >default (no copyfield tags)
>>>> >
>>>> >success
>>>> >
>>>> >default plus only jar references
>>>> >
>>>> >modified with one copyfield tag
>>>> >
>>>> >success
>>>> >
>>>> >default plus only jar references
>>>> >
>>>> >modified with two copyfield tags
>>>> >
>>>> >crash
>>>> >
>>>> >additional modified settings
>>>> >
>>>> >default (no copyfield tags)
>>>> >
>>>> >success
>>>> >
>>>> >additional modified settings
>>>> >
>>>> >modified with one copyfield tag
>>>> >
>>>> >success
>>>> >
>>>> >additional modified settings
>>>> >
>>>> >modified with two copyfield tags
>>>> >
>>>> >crash
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >Our question is, what can we do to eliminate these OOM exceptions?
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
Does this user list allow attachments? I have four files attached
(database.xml, error.txt, schema.xml, solrconfig.xml). We just ran the
process again using the parameters you suggested, but not to a csv file.
It errored out quickly. We are working on the csv file run.
Removed both <autoCommit> and <autoSoftCommit> parts/definitions from
solrconfig.xml
Disabled tlog by removing
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
</updateLog>
from solrconfig.xml
Used commit=true parameter. ?commit=true&command=full-import
On Fri, Apr 4, 2014 at 3:29 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> This may not solve your problem but generally it is recommended to disable
> auto commit and transaction logs for bulk indexing.
> And issue one commit at the very end. Do you tlogs enabled? I see "commit
> failed" in the error message thats why I am offering this.
>
> And regarding comma separated values, with this approach you focus on just
> solr importing process. You separate data acquisition phrase. And it is
> very fast load even big csv files http://wiki.apache.org/solr/UpdateCSV
> I have never experienced OOM during indexing, I suspect data acquisition
> has role in it.
>
> Ahmet
>
> On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
> We would be happy to try that. That sounds counter intuitive for the high
> volume of records we have. Can you help me understand how that might solve
> our problem?
>
>
>
>
> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
> Hi,
> >
> >Can you remove auto commit for bulk import. Commit at the very end?
> >
> >Ahmet
> >
> >
> >
> >
> >On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >In case the attached database.xml file didn't show up, I have pasted in
> the
> >contents below:
> >
> ><dataConfig>
> ><dataSource
> >name="org_only"
> >type="JdbcDataSource"
> >driver="oracle.jdbc.OracleDriver"
> >url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
> >user="admin"
> >password="admin"
> >readOnly="false"
> >batchSize="100"
> >/>
> ><document>
> >
> >
> ><entity name="full-index" query="
> >select
> >
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
> >as SOLR_ID,
> >
> >'ORCL.ADDRESS_ACCT_ALL'
> >as SOLR_CATEGORY,
> >
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
> >ADDRESSALLROWID,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> >ADDRESSALLADDRTYPECD,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
> >ADDRESSALLLONGITUDE,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
> >ADDRESSALLLATITUDE,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
> >ADDRESSALLADDRNAME,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
> >ADDRESSALLCITY,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
> >ADDRESSALLSTATE,
> >NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> >ADDRESSALLEMAILADDR
> >
> >from ORCL.ADDRESS_ACCT_ALL
> >" >
> >
> ><field column="SOLR_ID" name="id" />
> ><field column="SOLR_CATEGORY" name="category" />
> ><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
> ><field column="ADDRESSALLADDRTYPECD"
> >name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
> ><field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc"
> />
> ><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
> ><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
> />
> ><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
> ><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
> ><field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
> >/>
> >
> ></entity>
> >
> >
> >
> ><!-- Varaibles -->
> ><!-- '${dataimporter.last_index_time}' -->
> ></document>
> ></dataConfig>
> >
> >
> >
> >
> >
> >
> >On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
> >candygram.for.mongo@gmail.com> wrote:
> >
> >> In this case we are indexing an Oracle database.
> >>
> >> We do not include the data-config.xml in our distribution. We store the
> >> database information in the database.xml file. I have attached the
> >> database.xml file.
> >>
> >> When we use the default merge policy settings, we get the same results.
> >>
> >>
> >>
> >> We have not tried to dump the table to a comma separated file. We think
> >> that dumping this size table to disk will introduce other memory
> problems
> >> with big file management. We have not tested that case.
> >>
> >>
> >> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Which database are you using? Can you send us data-config.xml?
> >>>
> >>> What happens when you use default merge policy settings?
> >>>
> >>> What happens when you dump your table to Comma Separated File and fed
> >>> that file to solr?
> >>>
> >>> Ahmet
> >>>
> >>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> >>> candygram.for.mongo@gmail.com> wrote:
> >>>
> >>> The ramBufferSizeMB was set to 6MB only on the test system to make the
> >>> system crash sooner. In production that tag is commented out which
> >>> I believe forces the default value to be used.
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
> wrote:
> >>>
> >>> Hi,
> >>> >
> >>> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >>> >
> >>> >Ahmet
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> >>> candygram.for.mongo@gmail.com> wrote:
> >>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
> Exception
> >>> >
> >>> >*SOLR/Lucene version: *4.2.1*
> >>> >
> >>> >
> >>> >*JVM version:
> >>> >
> >>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >>> >
> >>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >>> >
> >>> >
> >>> >
> >>> >*Indexer startup command:
> >>> >
> >>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >>> >
> >>> >
> >>> >
> >>> >java " %JVMARGS% ^
> >>> >
> >>> >-Dcom.sun.management.jmxremote.port=1092 ^
> >>> >
> >>> >-Dcom.sun.management.jmxremote.ssl=false ^
> >>> >
> >>> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >>> >
> >>> >-jar start.jar
> >>> >
> >>> >
> >>> >
> >>> >*SOLR indexing HTTP parameters request:
> >>> >
> >>> >webapp=/solr path=/dataimport
> >>> >params={clean=false&command=full-import&wt=javabin&version=2}
> >>> >
> >>> >
> >>> >
> >>> >We are getting a Java heap OOM exception when indexing (updating) 27
> >>> >million records. If we increase the Java heap memory settings the
> >>> problem
> >>> >goes away but we believe the problem has not been fixed and that we
> will
> >>> >eventually get the same OOM exception. We have other processes on the
> >>> >server that also require resources so we cannot continually increase
> the
> >>> >memory settings to resolve the OOM issue. We are trying to find a
> way to
> >>> >configure the SOLR instance to reduce or preferably eliminate the
> >>> >possibility of an OOM exception.
> >>> >
> >>> >
> >>> >
> >>> >We can reproduce the problem on a test machine. We set the Java heap
> >>> >memory size to 64MB to accelerate the exception. If we increase this
> >>> >setting the same problems occurs, just hours later. In the test
> >>> >environment, we are using the following parameters:
> >>> >
> >>> >
> >>> >
> >>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >>> >
> >>> >
> >>> >
> >>> >Normally we use the default solrconfig.xml file with only the
> following
> >>> jar
> >>> >file references added:
> >>> >
> >>> >
> >>> >
> >>> ><lib path="../../../../default/lib/common.jar" />
> >>> >
> >>> ><lib path="../../../../default/lib/webapp.jar" />
> >>> >
> >>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >>> >
> >>> >
> >>> >
> >>> >Using these values and trying to index 6 million records from the
> >>> database,
> >>> >the Java Heap Out of Memory exception is thrown very quickly.
> >>> >
> >>> >
> >>> >
> >>> >We were able to complete a successful indexing by further modifying
> the
> >>> >solrconfig.xml and removing all or all but one <copyfield> tags from
> the
> >>> >schema.xml file.
> >>> >
> >>> >
> >>> >
> >>> >The following solrconfig.xml values were modified:
> >>> >
> >>> >
> >>> >
> >>> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >>> >
> >>> >
> >>> >
> >>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >>> >
> >>> ><int name="maxMergeAtOnce">2</int>
> >>> >
> >>> ><int name="maxMergeAtOnceExplicit">2</int>
> >>> >
> >>> ><int name="segmentsPerTier">10</int>
> >>> >
> >>> ><int name="maxMergedSegmentMB">150</int>
> >>> >
> >>> ></mergePolicy>
> >>> >
> >>> >
> >>> >
> >>> ><autoCommit>
> >>> >
> >>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
> -- >
> >>> >
> >>> ><openSearcher>false</openSearcher>
> >>> >
> >>> ></autoCommit>
> >>> >
> >>> >
> >>> >
> >>> >Using our customized schema.xml file with two or more <copyfield>
> tags,
> >>> the
> >>> >OOM exception is always thrown. Based on the errors, the problem
> occurs
> >>> >when the process was trying to do the merge. The error is provided
> >>> below:
> >>> >
> >>> >
> >>> >
> >>> >Exception in thread "Lucene Merge Thread #156"
> >>> >org.apache.lucene.index.MergePolicy$MergeException:
> >>> >java.lang.OutOfMemoryError: Java heap space
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >>> >
> >>> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >>> >
> >>> > at
> >>>
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >>> >
> >>> > at
> >>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >>> >
> >>> > at
> >>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >>> >
> >>> > at
> >>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >>> >
> >>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >>> >
> >>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
> writer
> >>> >hit an OutOfMemoryError; cannot commit
> >>> >
> >>> > at
> >>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >>> >
> >>> > at
> >>>
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >>> >
> >>> > at
> >>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >>> >
> >>> > at
> >>>
> >>>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >>> >
> >>> > at
> >>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >>> >
> >>> > at
> >>>
> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>> >
> >>> > at
> >>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >>> >
> >>> > at
> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>> >
> >>> > at
> >>>
> >>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>> >
> >>> > at
> >>>
> >>>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>> >
> >>> > at
> >>>
> >>>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >>> >
> >>> > at
> >>>
> >>>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>> >
> >>> > at java.lang.Thread.run(Thread.java:722)
> >>> >
> >>> >
> >>> >
> >>> >We think but are not 100% sure that the problem is related to the
> merge.
> >>> >
> >>> >
> >>> >
> >>> >Normally our schema.xml contains a lot of field specifications (like
> the
> >>> >ones seen in the file fragment below):
> >>> >
> >>> >
> >>> >
> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>> dest="ADDRESS.RECORD_ID.case_abc"
> >>> >/>
> >>> >
> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >>> >
> >>> ><copyField source="ADDRESS.RECORD_ID_abc"
> >>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >>> >
> >>> >
> >>> >
> >>> >In tests using the default file schema.xml and no <copyfield> tags,
> >>> >indexing completed successfully. 6 million records produced a 900 MB
> >>> data
> >>> >directory.
> >>> >
> >>> >
> >>> >
> >>> >When I included just one <copyfield> tag, indexing completed
> >>> successfully. 6
> >>> >million records produced a 990 MB data directory (90 MB bigger).
> >>> >
> >>> >
> >>> >
> >>> >When I included just two <copyfield> tags, the index crashed with an
> OOM
> >>> >exception.
> >>> >
> >>> >
> >>> >
> >>> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
> >>> the
> >>> >crash.
> >>> >
> >>> >
> >>> >
> >>> >The net of our test results I as follows:
> >>> >
> >>> >
> >>> >
> >>> >*solrconfig.xml*
> >>> >
> >>> >*schema.xml*
> >>> >
> >>> >*result*
> >>> >
> >>> >
> >>> >default plus only jar references
> >>> >
> >>> >default (no copyfield tags)
> >>> >
> >>> >success
> >>> >
> >>> >default plus only jar references
> >>> >
> >>> >modified with one copyfield tag
> >>> >
> >>> >success
> >>> >
> >>> >default plus only jar references
> >>> >
> >>> >modified with two copyfield tags
> >>> >
> >>> >crash
> >>> >
> >>> >additional modified settings
> >>> >
> >>> >default (no copyfield tags)
> >>> >
> >>> >success
> >>> >
> >>> >additional modified settings
> >>> >
> >>> >modified with one copyfield tag
> >>> >
> >>> >success
> >>> >
> >>> >additional modified settings
> >>> >
> >>> >modified with two copyfield tags
> >>> >
> >>> >crash
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >Our question is, what can we do to eliminate these OOM exceptions?
> >>> >
> >>> >
> >>>
> >>
> >>
> >
> >
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
This may not solve your problem but generally it is recommended to disable auto commit and transaction logs for bulk indexing.
And issue one commit at the very end. Do you tlogs enabled? I see "commit failed" in the error message thats why I am offering this.
And regarding comma separated values, with this approach you focus on just solr importing process. You separate data acquisition phrase. And it is very fast load even big csv files http://wiki.apache.org/solr/UpdateCSV
I have never experienced OOM during indexing, I suspect data acquisition has role in it.
Ahmet
On Saturday, April 5, 2014 1:18 AM, Candygram For Mongo <ca...@gmail.com> wrote:
We would be happy to try that. That sounds counter intuitive for the high volume of records we have. Can you help me understand how that might solve our problem?
On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
Hi,
>
>Can you remove auto commit for bulk import. Commit at the very end?
>
>Ahmet
>
>
>
>
>On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <ca...@gmail.com> wrote:
>In case the attached database.xml file didn't show up, I have pasted in the
>contents below:
>
><dataConfig>
><dataSource
>name="org_only"
>type="JdbcDataSource"
>driver="oracle.jdbc.OracleDriver"
>url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>user="admin"
>password="admin"
>readOnly="false"
>batchSize="100"
>/>
><document>
>
>
><entity name="full-index" query="
>select
>
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>as SOLR_ID,
>
>'ORCL.ADDRESS_ACCT_ALL'
>as SOLR_CATEGORY,
>
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>ADDRESSALLROWID,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>ADDRESSALLADDRTYPECD,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>ADDRESSALLLONGITUDE,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>ADDRESSALLLATITUDE,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>ADDRESSALLADDRNAME,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>ADDRESSALLCITY,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>ADDRESSALLSTATE,
>NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>ADDRESSALLEMAILADDR
>
>from ORCL.ADDRESS_ACCT_ALL
>" >
>
><field column="SOLR_ID" name="id" />
><field column="SOLR_CATEGORY" name="category" />
><field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
><field column="ADDRESSALLADDRTYPECD"
>name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
><field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
><field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
><field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
><field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
><field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
><field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>/>
>
></entity>
>
>
>
><!-- Varaibles -->
><!-- '${dataimporter.last_index_time}' -->
></document>
></dataConfig>
>
>
>
>
>
>
>On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>candygram.for.mongo@gmail.com> wrote:
>
>> In this case we are indexing an Oracle database.
>>
>> We do not include the data-config.xml in our distribution. We store the
>> database information in the database.xml file. I have attached the
>> database.xml file.
>>
>> When we use the default merge policy settings, we get the same results.
>>
>>
>>
>> We have not tried to dump the table to a comma separated file. We think
>> that dumping this size table to disk will introduce other memory problems
>> with big file management. We have not tested that case.
>>
>>
>> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>>
>>> Hi,
>>>
>>> Which database are you using? Can you send us data-config.xml?
>>>
>>> What happens when you use default merge policy settings?
>>>
>>> What happens when you dump your table to Comma Separated File and fed
>>> that file to solr?
>>>
>>> Ahmet
>>>
>>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>>> candygram.for.mongo@gmail.com> wrote:
>>>
>>> The ramBufferSizeMB was set to 6MB only on the test system to make the
>>> system crash sooner. In production that tag is commented out which
>>> I believe forces the default value to be used.
>>>
>>>
>>>
>>>
>>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>>
>>> Hi,
>>> >
>>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>>> >
>>> >Ahmet
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>>> candygram.for.mongo@gmail.com> wrote:
>>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>>> >
>>> >*SOLR/Lucene version: *4.2.1*
>>> >
>>> >
>>> >*JVM version:
>>> >
>>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>>> >
>>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>>> >
>>> >
>>> >
>>> >*Indexer startup command:
>>> >
>>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>>> >
>>> >
>>> >
>>> >java " %JVMARGS% ^
>>> >
>>> >-Dcom.sun.management.jmxremote.port=1092 ^
>>> >
>>> >-Dcom.sun.management.jmxremote.ssl=false ^
>>> >
>>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>>> >
>>> >-jar start.jar
>>> >
>>> >
>>> >
>>> >*SOLR indexing HTTP parameters request:
>>> >
>>> >webapp=/solr path=/dataimport
>>> >params={clean=false&command=full-import&wt=javabin&version=2}
>>> >
>>> >
>>> >
>>> >We are getting a Java heap OOM exception when indexing (updating) 27
>>> >million records. If we increase the Java heap memory settings the
>>> problem
>>> >goes away but we believe the problem has not been fixed and that we will
>>> >eventually get the same OOM exception. We have other processes on the
>>> >server that also require resources so we cannot continually increase the
>>> >memory settings to resolve the OOM issue. We are trying to find a way to
>>> >configure the SOLR instance to reduce or preferably eliminate the
>>> >possibility of an OOM exception.
>>> >
>>> >
>>> >
>>> >We can reproduce the problem on a test machine. We set the Java heap
>>> >memory size to 64MB to accelerate the exception. If we increase this
>>> >setting the same problems occurs, just hours later. In the test
>>> >environment, we are using the following parameters:
>>> >
>>> >
>>> >
>>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>>> >
>>> >
>>> >
>>> >Normally we use the default solrconfig.xml file with only the following
>>> jar
>>> >file references added:
>>> >
>>> >
>>> >
>>> ><lib path="../../../../default/lib/common.jar" />
>>> >
>>> ><lib path="../../../../default/lib/webapp.jar" />
>>> >
>>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>>> >
>>> >
>>> >
>>> >Using these values and trying to index 6 million records from the
>>> database,
>>> >the Java Heap Out of Memory exception is thrown very quickly.
>>> >
>>> >
>>> >
>>> >We were able to complete a successful indexing by further modifying the
>>> >solrconfig.xml and removing all or all but one <copyfield> tags from the
>>> >schema.xml file.
>>> >
>>> >
>>> >
>>> >The following solrconfig.xml values were modified:
>>> >
>>> >
>>> >
>>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>>> >
>>> >
>>> >
>>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>> >
>>> ><int name="maxMergeAtOnce">2</int>
>>> >
>>> ><int name="maxMergeAtOnceExplicit">2</int>
>>> >
>>> ><int name="segmentsPerTier">10</int>
>>> >
>>> ><int name="maxMergedSegmentMB">150</int>
>>> >
>>> ></mergePolicy>
>>> >
>>> >
>>> >
>>> ><autoCommit>
>>> >
>>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>>> >
>>> ><openSearcher>false</openSearcher>
>>> >
>>> ></autoCommit>
>>> >
>>> >
>>> >
>>> >Using our customized schema.xml file with two or more <copyfield> tags,
>>> the
>>> >OOM exception is always thrown. Based on the errors, the problem occurs
>>> >when the process was trying to do the merge. The error is provided
>>> below:
>>> >
>>> >
>>> >
>>> >Exception in thread "Lucene Merge Thread #156"
>>> >org.apache.lucene.index.MergePolicy$MergeException:
>>> >java.lang.OutOfMemoryError: Java heap space
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>> >
>>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>>> >
>>> > at
>>>
>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>>> >
>>> > at
>>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>>> >
>>> > at
>>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>>> >
>>> > at
>>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>>> >
>>> > at
>>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>> >
>>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>>> >
>>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>>> >hit an OutOfMemoryError; cannot commit
>>> >
>>> > at
>>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>>> >
>>> > at
>>>
>>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>>> >
>>> > at
>>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>>> >
>>> > at
>>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>>> >
>>> > at
>>>
>>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>>> >
>>> > at
>>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>>> >
>>> > at
>>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> >
>>> > at
>>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >
>>> > at
>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >
>>> > at
>>>
>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>> >
>>> > at
>>>
>>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>> >
>>> > at
>>>
>>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> >
>>> > at
>>>
>>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> >
>>> > at java.lang.Thread.run(Thread.java:722)
>>> >
>>> >
>>> >
>>> >We think but are not 100% sure that the problem is related to the merge.
>>> >
>>> >
>>> >
>>> >Normally our schema.xml contains a lot of field specifications (like the
>>> >ones seen in the file fragment below):
>>> >
>>> >
>>> >
>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> dest="ADDRESS.RECORD_ID.case_abc"
>>> >/>
>>> >
>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>>> >
>>> ><copyField source="ADDRESS.RECORD_ID_abc"
>>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>>> >
>>> >
>>> >
>>> >In tests using the default file schema.xml and no <copyfield> tags,
>>> >indexing completed successfully. 6 million records produced a 900 MB
>>> data
>>> >directory.
>>> >
>>> >
>>> >
>>> >When I included just one <copyfield> tag, indexing completed
>>> successfully. 6
>>> >million records produced a 990 MB data directory (90 MB bigger).
>>> >
>>> >
>>> >
>>> >When I included just two <copyfield> tags, the index crashed with an OOM
>>> >exception.
>>> >
>>> >
>>> >
>>> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>>> the
>>> >crash.
>>> >
>>> >
>>> >
>>> >The net of our test results I as follows:
>>> >
>>> >
>>> >
>>> >*solrconfig.xml*
>>> >
>>> >*schema.xml*
>>> >
>>> >*result*
>>> >
>>> >
>>> >default plus only jar references
>>> >
>>> >default (no copyfield tags)
>>> >
>>> >success
>>> >
>>> >default plus only jar references
>>> >
>>> >modified with one copyfield tag
>>> >
>>> >success
>>> >
>>> >default plus only jar references
>>> >
>>> >modified with two copyfield tags
>>> >
>>> >crash
>>> >
>>> >additional modified settings
>>> >
>>> >default (no copyfield tags)
>>> >
>>> >success
>>> >
>>> >additional modified settings
>>> >
>>> >modified with one copyfield tag
>>> >
>>> >success
>>> >
>>> >additional modified settings
>>> >
>>> >modified with two copyfield tags
>>> >
>>> >crash
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >Our question is, what can we do to eliminate these OOM exceptions?
>>> >
>>> >
>>>
>>
>>
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
To disable auto commit remove both <autoCommit> and <autoSoftCommit> parts/definitions from solrconfig.xml
To disable tlog remove
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
</updateLog>
from solrconfig.xml
To commit at the end use commit=true parameter. ?commit=true&command=full-import
There is a checkbox for this in data import admin page.
On Saturday, April 5, 2014 1:27 AM, Candygram For Mongo <ca...@gmail.com> wrote:
I might have forgot to mention that we are using the DataImportHandler. I
think we know how to remove auto commit. How would we force a commit at
the end?
On Fri, Apr 4, 2014 at 3:18 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> We would be happy to try that. That sounds counter intuitive for the high
> volume of records we have. Can you help me understand how that might solve
> our problem?
>
>
>
> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Hi,
>>
>> Can you remove auto commit for bulk import. Commit at the very end?
>>
>> Ahmet
>>
>>
>>
>> On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>> In case the attached database.xml file didn't show up, I have pasted in
>> the
>> contents below:
>>
>> <dataConfig>
>> <dataSource
>> name="org_only"
>> type="JdbcDataSource"
>> driver="oracle.jdbc.OracleDriver"
>> url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>> user="admin"
>> password="admin"
>> readOnly="false"
>> batchSize="100"
>> />
>> <document>
>>
>>
>> <entity name="full-index" query="
>> select
>>
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>> as SOLR_ID,
>>
>> 'ORCL.ADDRESS_ACCT_ALL'
>> as SOLR_CATEGORY,
>>
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>> ADDRESSALLROWID,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>> ADDRESSALLADDRTYPECD,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLONGITUDE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLATITUDE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>> ADDRESSALLADDRNAME,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>> ADDRESSALLCITY,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>> ADDRESSALLSTATE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>> ADDRESSALLEMAILADDR
>>
>> from ORCL.ADDRESS_ACCT_ALL
>> " >
>>
>> <field column="SOLR_ID" name="id" />
>> <field column="SOLR_CATEGORY" name="category" />
>> <field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
>> <field column="ADDRESSALLADDRTYPECD"
>> name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc"
>> />
>> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
>> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
>> />
>> <field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>> <field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>> />
>>
>> </entity>
>>
>>
>>
>> <!-- Varaibles -->
>> <!-- '${dataimporter.last_index_time}' -->
>> </document>
>> </dataConfig>
>>
>>
>>
>>
>>
>>
>> On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>> > In this case we are indexing an Oracle database.
>> >
>> > We do not include the data-config.xml in our distribution. We store the
>> > database information in the database.xml file. I have attached the
>> > database.xml file.
>> >
>> > When we use the default merge policy settings, we get the same results.
>> >
>> >
>> >
>> > We have not tried to dump the table to a comma separated file. We think
>> > that dumping this size table to disk will introduce other memory
>> problems
>> > with big file management. We have not tested that case.
>> >
>> >
>> > On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Which database are you using? Can you send us data-config.xml?
>> >>
>> >> What happens when you use default merge policy settings?
>> >>
>> >> What happens when you dump your table to Comma Separated File and fed
>> >> that file to solr?
>> >>
>> >> Ahmet
>> >>
>> >> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>> >> candygram.for.mongo@gmail.com> wrote:
>> >>
>> >> The ramBufferSizeMB was set to 6MB only on the test system to make the
>> >> system crash sooner. In production that tag is commented out which
>> >> I believe forces the default value to be used.
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>> wrote:
>> >>
>> >> Hi,
>> >> >
>> >> >out of curiosity, why did you set ramBufferSizeMB to 6?
>> >> >
>> >> >Ahmet
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>> >> candygram.for.mongo@gmail.com> wrote:
>> >> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>> Exception
>> >> >
>> >> >*SOLR/Lucene version: *4.2.1*
>> >> >
>> >> >
>> >> >*JVM version:
>> >> >
>> >> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>> >> >
>> >> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>> >> >
>> >> >
>> >> >
>> >> >*Indexer startup command:
>> >> >
>> >> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>> >> >
>> >> >
>> >> >
>> >> >java " %JVMARGS% ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.port=1092 ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.ssl=false ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.authenticate=false ^
>> >> >
>> >> >-jar start.jar
>> >> >
>> >> >
>> >> >
>> >> >*SOLR indexing HTTP parameters request:
>> >> >
>> >> >webapp=/solr path=/dataimport
>> >> >params={clean=false&command=full-import&wt=javabin&version=2}
>> >> >
>> >> >
>> >> >
>> >> >We are getting a Java heap OOM exception when indexing (updating) 27
>> >> >million records. If we increase the Java heap memory settings the
>> >> problem
>> >> >goes away but we believe the problem has not been fixed and that we
>> will
>> >> >eventually get the same OOM exception. We have other processes on the
>> >> >server that also require resources so we cannot continually increase
>> the
>> >> >memory settings to resolve the OOM issue. We are trying to find a
>> way to
>> >> >configure the SOLR instance to reduce or preferably eliminate the
>> >> >possibility of an OOM exception.
>> >> >
>> >> >
>> >> >
>> >> >We can reproduce the problem on a test machine. We set the Java heap
>> >> >memory size to 64MB to accelerate the exception. If we increase this
>> >> >setting the same problems occurs, just hours later. In the test
>> >> >environment, we are using the following parameters:
>> >> >
>> >> >
>> >> >
>> >> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> >> >
>> >> >
>> >> >
>> >> >Normally we use the default solrconfig.xml file with only the
>> following
>> >> jar
>> >> >file references added:
>> >> >
>> >> >
>> >> >
>> >> ><lib path="../../../../default/lib/common.jar" />
>> >> >
>> >> ><lib path="../../../../default/lib/webapp.jar" />
>> >> >
>> >> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>> >> >
>> >> >
>> >> >
>> >> >Using these values and trying to index 6 million records from the
>> >> database,
>> >> >the Java Heap Out of Memory exception is thrown very quickly.
>> >> >
>> >> >
>> >> >
>> >> >We were able to complete a successful indexing by further modifying
>> the
>> >> >solrconfig.xml and removing all or all but one <copyfield> tags from
>> the
>> >> >schema.xml file.
>> >> >
>> >> >
>> >> >
>> >> >The following solrconfig.xml values were modified:
>> >> >
>> >> >
>> >> >
>> >> ><ramBufferSizeMB>6</ramBufferSizeMB>
>> >> >
>> >> >
>> >> >
>> >> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> >> >
>> >> ><int name="maxMergeAtOnce">2</int>
>> >> >
>> >> ><int name="maxMergeAtOnceExplicit">2</int>
>> >> >
>> >> ><int name="segmentsPerTier">10</int>
>> >> >
>> >> ><int name="maxMergedSegmentMB">150</int>
>> >> >
>> >> ></mergePolicy>
>> >> >
>> >> >
>> >> >
>> >> ><autoCommit>
>> >> >
>> >> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
>> -- >
>> >> >
>> >> ><openSearcher>false</openSearcher>
>> >> >
>> >> ></autoCommit>
>> >> >
>> >> >
>> >> >
>> >> >Using our customized schema.xml file with two or more <copyfield>
>> tags,
>> >> the
>> >> >OOM exception is always thrown. Based on the errors, the problem
>> occurs
>> >> >when the process was trying to do the merge. The error is provided
>> >> below:
>> >> >
>> >> >
>> >> >
>> >> >Exception in thread "Lucene Merge Thread #156"
>> >> >org.apache.lucene.index.MergePolicy$MergeException:
>> >> >java.lang.OutOfMemoryError: Java heap space
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> >> >
>> >> >Caused by: java.lang.OutOfMemoryError: Java heap space
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>> >> >
>> >> > at
>> >>
>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> >> >
>> >> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>> >> >
>> >> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
>> writer
>> >> >hit an OutOfMemoryError; cannot commit
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>> >> >
>> >> > at
>> >>
>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>> >> >
>> >> > at
>> >> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>> >> >
>> >> > at
>> >>
>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >> >
>> >> > at
>> >> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >> >
>> >> > at
>> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:722)
>> >> >
>> >> >
>> >> >
>> >> >We think but are not 100% sure that the problem is related to the
>> merge.
>> >> >
>> >> >
>> >> >
>> >> >Normally our schema.xml contains a lot of field specifications (like
>> the
>> >> >ones seen in the file fragment below):
>> >> >
>> >> >
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> dest="ADDRESS.RECORD_ID.case_abc"
>> >> >/>
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>> >> >
>> >> >
>> >> >
>> >> >In tests using the default file schema.xml and no <copyfield> tags,
>> >> >indexing completed successfully. 6 million records produced a 900 MB
>> >> data
>> >> >directory.
>> >> >
>> >> >
>> >> >
>> >> >When I included just one <copyfield> tag, indexing completed
>> >> successfully. 6
>> >> >million records produced a 990 MB data directory (90 MB bigger).
>> >> >
>> >> >
>> >> >
>> >> >When I included just two <copyfield> tags, the index crashed with an
>> OOM
>> >> >exception.
>> >> >
>> >> >
>> >> >
>> >> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>> >> the
>> >> >crash.
>> >> >
>> >> >
>> >> >
>> >> >The net of our test results I as follows:
>> >> >
>> >> >
>> >> >
>> >> >*solrconfig.xml*
>> >> >
>> >> >*schema.xml*
>> >> >
>> >> >*result*
>> >> >
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >default (no copyfield tags)
>> >> >
>> >> >success
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >modified with one copyfield tag
>> >> >
>> >> >success
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >modified with two copyfield tags
>> >> >
>> >> >crash
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >default (no copyfield tags)
>> >> >
>> >> >success
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >modified with one copyfield tag
>> >> >
>> >> >success
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >modified with two copyfield tags
>> >> >
>> >> >crash
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >Our question is, what can we do to eliminate these OOM exceptions?
>> >> >
>> >> >
>> >>
>> >
>> >
>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
I might have forgot to mention that we are using the DataImportHandler. I
think we know how to remove auto commit. How would we force a commit at
the end?
On Fri, Apr 4, 2014 at 3:18 PM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> We would be happy to try that. That sounds counter intuitive for the high
> volume of records we have. Can you help me understand how that might solve
> our problem?
>
>
>
> On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Hi,
>>
>> Can you remove auto commit for bulk import. Commit at the very end?
>>
>> Ahmet
>>
>>
>>
>> On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>> In case the attached database.xml file didn't show up, I have pasted in
>> the
>> contents below:
>>
>> <dataConfig>
>> <dataSource
>> name="org_only"
>> type="JdbcDataSource"
>> driver="oracle.jdbc.OracleDriver"
>> url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
>> user="admin"
>> password="admin"
>> readOnly="false"
>> batchSize="100"
>> />
>> <document>
>>
>>
>> <entity name="full-index" query="
>> select
>>
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
>> as SOLR_ID,
>>
>> 'ORCL.ADDRESS_ACCT_ALL'
>> as SOLR_CATEGORY,
>>
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
>> ADDRESSALLROWID,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
>> ADDRESSALLADDRTYPECD,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLONGITUDE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
>> ADDRESSALLLATITUDE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
>> ADDRESSALLADDRNAME,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
>> ADDRESSALLCITY,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
>> ADDRESSALLSTATE,
>> NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
>> ADDRESSALLEMAILADDR
>>
>> from ORCL.ADDRESS_ACCT_ALL
>> " >
>>
>> <field column="SOLR_ID" name="id" />
>> <field column="SOLR_CATEGORY" name="category" />
>> <field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
>> <field column="ADDRESSALLADDRTYPECD"
>> name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
>> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc"
>> />
>> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
>> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc"
>> />
>> <field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
>> <field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
>> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
>> />
>>
>> </entity>
>>
>>
>>
>> <!-- Varaibles -->
>> <!-- '${dataimporter.last_index_time}' -->
>> </document>
>> </dataConfig>
>>
>>
>>
>>
>>
>>
>> On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>> > In this case we are indexing an Oracle database.
>> >
>> > We do not include the data-config.xml in our distribution. We store the
>> > database information in the database.xml file. I have attached the
>> > database.xml file.
>> >
>> > When we use the default merge policy settings, we get the same results.
>> >
>> >
>> >
>> > We have not tried to dump the table to a comma separated file. We think
>> > that dumping this size table to disk will introduce other memory
>> problems
>> > with big file management. We have not tested that case.
>> >
>> >
>> > On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Which database are you using? Can you send us data-config.xml?
>> >>
>> >> What happens when you use default merge policy settings?
>> >>
>> >> What happens when you dump your table to Comma Separated File and fed
>> >> that file to solr?
>> >>
>> >> Ahmet
>> >>
>> >> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>> >> candygram.for.mongo@gmail.com> wrote:
>> >>
>> >> The ramBufferSizeMB was set to 6MB only on the test system to make the
>> >> system crash sooner. In production that tag is commented out which
>> >> I believe forces the default value to be used.
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com>
>> wrote:
>> >>
>> >> Hi,
>> >> >
>> >> >out of curiosity, why did you set ramBufferSizeMB to 6?
>> >> >
>> >> >Ahmet
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>> >> candygram.for.mongo@gmail.com> wrote:
>> >> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
>> Exception
>> >> >
>> >> >*SOLR/Lucene version: *4.2.1*
>> >> >
>> >> >
>> >> >*JVM version:
>> >> >
>> >> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>> >> >
>> >> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>> >> >
>> >> >
>> >> >
>> >> >*Indexer startup command:
>> >> >
>> >> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>> >> >
>> >> >
>> >> >
>> >> >java " %JVMARGS% ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.port=1092 ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.ssl=false ^
>> >> >
>> >> >-Dcom.sun.management.jmxremote.authenticate=false ^
>> >> >
>> >> >-jar start.jar
>> >> >
>> >> >
>> >> >
>> >> >*SOLR indexing HTTP parameters request:
>> >> >
>> >> >webapp=/solr path=/dataimport
>> >> >params={clean=false&command=full-import&wt=javabin&version=2}
>> >> >
>> >> >
>> >> >
>> >> >We are getting a Java heap OOM exception when indexing (updating) 27
>> >> >million records. If we increase the Java heap memory settings the
>> >> problem
>> >> >goes away but we believe the problem has not been fixed and that we
>> will
>> >> >eventually get the same OOM exception. We have other processes on the
>> >> >server that also require resources so we cannot continually increase
>> the
>> >> >memory settings to resolve the OOM issue. We are trying to find a
>> way to
>> >> >configure the SOLR instance to reduce or preferably eliminate the
>> >> >possibility of an OOM exception.
>> >> >
>> >> >
>> >> >
>> >> >We can reproduce the problem on a test machine. We set the Java heap
>> >> >memory size to 64MB to accelerate the exception. If we increase this
>> >> >setting the same problems occurs, just hours later. In the test
>> >> >environment, we are using the following parameters:
>> >> >
>> >> >
>> >> >
>> >> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> >> >
>> >> >
>> >> >
>> >> >Normally we use the default solrconfig.xml file with only the
>> following
>> >> jar
>> >> >file references added:
>> >> >
>> >> >
>> >> >
>> >> ><lib path="../../../../default/lib/common.jar" />
>> >> >
>> >> ><lib path="../../../../default/lib/webapp.jar" />
>> >> >
>> >> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>> >> >
>> >> >
>> >> >
>> >> >Using these values and trying to index 6 million records from the
>> >> database,
>> >> >the Java Heap Out of Memory exception is thrown very quickly.
>> >> >
>> >> >
>> >> >
>> >> >We were able to complete a successful indexing by further modifying
>> the
>> >> >solrconfig.xml and removing all or all but one <copyfield> tags from
>> the
>> >> >schema.xml file.
>> >> >
>> >> >
>> >> >
>> >> >The following solrconfig.xml values were modified:
>> >> >
>> >> >
>> >> >
>> >> ><ramBufferSizeMB>6</ramBufferSizeMB>
>> >> >
>> >> >
>> >> >
>> >> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> >> >
>> >> ><int name="maxMergeAtOnce">2</int>
>> >> >
>> >> ><int name="maxMergeAtOnceExplicit">2</int>
>> >> >
>> >> ><int name="segmentsPerTier">10</int>
>> >> >
>> >> ><int name="maxMergedSegmentMB">150</int>
>> >> >
>> >> ></mergePolicy>
>> >> >
>> >> >
>> >> >
>> >> ><autoCommit>
>> >> >
>> >> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
>> -- >
>> >> >
>> >> ><openSearcher>false</openSearcher>
>> >> >
>> >> ></autoCommit>
>> >> >
>> >> >
>> >> >
>> >> >Using our customized schema.xml file with two or more <copyfield>
>> tags,
>> >> the
>> >> >OOM exception is always thrown. Based on the errors, the problem
>> occurs
>> >> >when the process was trying to do the merge. The error is provided
>> >> below:
>> >> >
>> >> >
>> >> >
>> >> >Exception in thread "Lucene Merge Thread #156"
>> >> >org.apache.lucene.index.MergePolicy$MergeException:
>> >> >java.lang.OutOfMemoryError: Java heap space
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> >> >
>> >> >Caused by: java.lang.OutOfMemoryError: Java heap space
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>> >> >
>> >> > at
>> >>
>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> >> >
>> >> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>> >> >
>> >> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
>> writer
>> >> >hit an OutOfMemoryError; cannot commit
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>> >> >
>> >> > at
>> >>
>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>> >> >
>> >> > at
>> >> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>> >> >
>> >> > at
>> >>
>> >>
>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>> >> >
>> >> > at
>> >> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>> >> >
>> >> > at
>> >>
>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >> >
>> >> > at
>> >> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >> >
>> >> > at
>> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >> >
>> >> > at
>> >>
>> >>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:722)
>> >> >
>> >> >
>> >> >
>> >> >We think but are not 100% sure that the problem is related to the
>> merge.
>> >> >
>> >> >
>> >> >
>> >> >Normally our schema.xml contains a lot of field specifications (like
>> the
>> >> >ones seen in the file fragment below):
>> >> >
>> >> >
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> dest="ADDRESS.RECORD_ID.case_abc"
>> >> >/>
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>> >> >
>> >> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>> >> >
>> >> >
>> >> >
>> >> >In tests using the default file schema.xml and no <copyfield> tags,
>> >> >indexing completed successfully. 6 million records produced a 900 MB
>> >> data
>> >> >directory.
>> >> >
>> >> >
>> >> >
>> >> >When I included just one <copyfield> tag, indexing completed
>> >> successfully. 6
>> >> >million records produced a 990 MB data directory (90 MB bigger).
>> >> >
>> >> >
>> >> >
>> >> >When I included just two <copyfield> tags, the index crashed with an
>> OOM
>> >> >exception.
>> >> >
>> >> >
>> >> >
>> >> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>> >> the
>> >> >crash.
>> >> >
>> >> >
>> >> >
>> >> >The net of our test results I as follows:
>> >> >
>> >> >
>> >> >
>> >> >*solrconfig.xml*
>> >> >
>> >> >*schema.xml*
>> >> >
>> >> >*result*
>> >> >
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >default (no copyfield tags)
>> >> >
>> >> >success
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >modified with one copyfield tag
>> >> >
>> >> >success
>> >> >
>> >> >default plus only jar references
>> >> >
>> >> >modified with two copyfield tags
>> >> >
>> >> >crash
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >default (no copyfield tags)
>> >> >
>> >> >success
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >modified with one copyfield tag
>> >> >
>> >> >success
>> >> >
>> >> >additional modified settings
>> >> >
>> >> >modified with two copyfield tags
>> >> >
>> >> >crash
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >Our question is, what can we do to eliminate these OOM exceptions?
>> >> >
>> >> >
>> >>
>> >
>> >
>>
>>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
We would be happy to try that. That sounds counter intuitive for the high
volume of records we have. Can you help me understand how that might solve
our problem?
On Fri, Apr 4, 2014 at 2:34 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> Can you remove auto commit for bulk import. Commit at the very end?
>
> Ahmet
>
>
>
> On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> In case the attached database.xml file didn't show up, I have pasted in the
> contents below:
>
> <dataConfig>
> <dataSource
> name="org_only"
> type="JdbcDataSource"
> driver="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
> user="admin"
> password="admin"
> readOnly="false"
> batchSize="100"
> />
> <document>
>
>
> <entity name="full-index" query="
> select
>
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
> as SOLR_ID,
>
> 'ORCL.ADDRESS_ACCT_ALL'
> as SOLR_CATEGORY,
>
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
> ADDRESSALLROWID,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
> ADDRESSALLADDRTYPECD,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
> ADDRESSALLLONGITUDE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
> ADDRESSALLLATITUDE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
> ADDRESSALLADDRNAME,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
> ADDRESSALLCITY,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
> ADDRESSALLSTATE,
> NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> ADDRESSALLEMAILADDR
>
> from ORCL.ADDRESS_ACCT_ALL
> " >
>
> <field column="SOLR_ID" name="id" />
> <field column="SOLR_CATEGORY" name="category" />
> <field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
> <field column="ADDRESSALLADDRTYPECD"
> name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
> <field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc"
> />
> <field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
> <field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
> <field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
> <field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
> <field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
> />
>
> </entity>
>
>
>
> <!-- Varaibles -->
> <!-- '${dataimporter.last_index_time}' -->
> </document>
> </dataConfig>
>
>
>
>
>
>
> On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
> > In this case we are indexing an Oracle database.
> >
> > We do not include the data-config.xml in our distribution. We store the
> > database information in the database.xml file. I have attached the
> > database.xml file.
> >
> > When we use the default merge policy settings, we get the same results.
> >
> >
> >
> > We have not tried to dump the table to a comma separated file. We think
> > that dumping this size table to disk will introduce other memory problems
> > with big file management. We have not tested that case.
> >
> >
> > On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
> >
> >> Hi,
> >>
> >> Which database are you using? Can you send us data-config.xml?
> >>
> >> What happens when you use default merge policy settings?
> >>
> >> What happens when you dump your table to Comma Separated File and fed
> >> that file to solr?
> >>
> >> Ahmet
> >>
> >> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> >> candygram.for.mongo@gmail.com> wrote:
> >>
> >> The ramBufferSizeMB was set to 6MB only on the test system to make the
> >> system crash sooner. In production that tag is commented out which
> >> I believe forces the default value to be used.
> >>
> >>
> >>
> >>
> >> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> >>
> >> Hi,
> >> >
> >> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >> >
> >> >Ahmet
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> >> candygram.for.mongo@gmail.com> wrote:
> >> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory
> Exception
> >> >
> >> >*SOLR/Lucene version: *4.2.1*
> >> >
> >> >
> >> >*JVM version:
> >> >
> >> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >> >
> >> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >> >
> >> >
> >> >
> >> >*Indexer startup command:
> >> >
> >> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >> >
> >> >
> >> >
> >> >java " %JVMARGS% ^
> >> >
> >> >-Dcom.sun.management.jmxremote.port=1092 ^
> >> >
> >> >-Dcom.sun.management.jmxremote.ssl=false ^
> >> >
> >> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >> >
> >> >-jar start.jar
> >> >
> >> >
> >> >
> >> >*SOLR indexing HTTP parameters request:
> >> >
> >> >webapp=/solr path=/dataimport
> >> >params={clean=false&command=full-import&wt=javabin&version=2}
> >> >
> >> >
> >> >
> >> >We are getting a Java heap OOM exception when indexing (updating) 27
> >> >million records. If we increase the Java heap memory settings the
> >> problem
> >> >goes away but we believe the problem has not been fixed and that we
> will
> >> >eventually get the same OOM exception. We have other processes on the
> >> >server that also require resources so we cannot continually increase
> the
> >> >memory settings to resolve the OOM issue. We are trying to find a way
> to
> >> >configure the SOLR instance to reduce or preferably eliminate the
> >> >possibility of an OOM exception.
> >> >
> >> >
> >> >
> >> >We can reproduce the problem on a test machine. We set the Java heap
> >> >memory size to 64MB to accelerate the exception. If we increase this
> >> >setting the same problems occurs, just hours later. In the test
> >> >environment, we are using the following parameters:
> >> >
> >> >
> >> >
> >> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >> >
> >> >
> >> >
> >> >Normally we use the default solrconfig.xml file with only the following
> >> jar
> >> >file references added:
> >> >
> >> >
> >> >
> >> ><lib path="../../../../default/lib/common.jar" />
> >> >
> >> ><lib path="../../../../default/lib/webapp.jar" />
> >> >
> >> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >> >
> >> >
> >> >
> >> >Using these values and trying to index 6 million records from the
> >> database,
> >> >the Java Heap Out of Memory exception is thrown very quickly.
> >> >
> >> >
> >> >
> >> >We were able to complete a successful indexing by further modifying the
> >> >solrconfig.xml and removing all or all but one <copyfield> tags from
> the
> >> >schema.xml file.
> >> >
> >> >
> >> >
> >> >The following solrconfig.xml values were modified:
> >> >
> >> >
> >> >
> >> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >> >
> >> >
> >> >
> >> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >> >
> >> ><int name="maxMergeAtOnce">2</int>
> >> >
> >> ><int name="maxMergeAtOnceExplicit">2</int>
> >> >
> >> ><int name="segmentsPerTier">10</int>
> >> >
> >> ><int name="maxMergedSegmentMB">150</int>
> >> >
> >> ></mergePolicy>
> >> >
> >> >
> >> >
> >> ><autoCommit>
> >> >
> >> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this
> -- >
> >> >
> >> ><openSearcher>false</openSearcher>
> >> >
> >> ></autoCommit>
> >> >
> >> >
> >> >
> >> >Using our customized schema.xml file with two or more <copyfield> tags,
> >> the
> >> >OOM exception is always thrown. Based on the errors, the problem
> occurs
> >> >when the process was trying to do the merge. The error is provided
> >> below:
> >> >
> >> >
> >> >
> >> >Exception in thread "Lucene Merge Thread #156"
> >> >org.apache.lucene.index.MergePolicy$MergeException:
> >> >java.lang.OutOfMemoryError: Java heap space
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >> >
> >> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >> >
> >> > at
> >>
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >> >
> >> > at
> >> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >> >
> >> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >> >
> >> >SEVERE: auto commit error...:java.lang.IllegalStateException: this
> writer
> >> >hit an OutOfMemoryError; cannot commit
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >> >
> >> > at
> >>
> >>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >> >
> >> > at
> >>
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >> >
> >> > at
> >> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >> >
> >> > at
> >>
> >>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >> >
> >> > at
> >> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >> >
> >> > at
> >> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> >
> >> > at
> >> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> >
> >> > at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >> >
> >> > at
> >>
> >>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >> >
> >> > at java.lang.Thread.run(Thread.java:722)
> >> >
> >> >
> >> >
> >> >We think but are not 100% sure that the problem is related to the
> merge.
> >> >
> >> >
> >> >
> >> >Normally our schema.xml contains a lot of field specifications (like
> the
> >> >ones seen in the file fragment below):
> >> >
> >> >
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> dest="ADDRESS.RECORD_ID.case_abc"
> >> >/>
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >> >
> >> ><copyField source="ADDRESS.RECORD_ID_abc"
> >> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >> >
> >> >
> >> >
> >> >In tests using the default file schema.xml and no <copyfield> tags,
> >> >indexing completed successfully. 6 million records produced a 900 MB
> >> data
> >> >directory.
> >> >
> >> >
> >> >
> >> >When I included just one <copyfield> tag, indexing completed
> >> successfully. 6
> >> >million records produced a 990 MB data directory (90 MB bigger).
> >> >
> >> >
> >> >
> >> >When I included just two <copyfield> tags, the index crashed with an
> OOM
> >> >exception.
> >> >
> >> >
> >> >
> >> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
> >> the
> >> >crash.
> >> >
> >> >
> >> >
> >> >The net of our test results I as follows:
> >> >
> >> >
> >> >
> >> >*solrconfig.xml*
> >> >
> >> >*schema.xml*
> >> >
> >> >*result*
> >> >
> >> >
> >> >default plus only jar references
> >> >
> >> >default (no copyfield tags)
> >> >
> >> >success
> >> >
> >> >default plus only jar references
> >> >
> >> >modified with one copyfield tag
> >> >
> >> >success
> >> >
> >> >default plus only jar references
> >> >
> >> >modified with two copyfield tags
> >> >
> >> >crash
> >> >
> >> >additional modified settings
> >> >
> >> >default (no copyfield tags)
> >> >
> >> >success
> >> >
> >> >additional modified settings
> >> >
> >> >modified with one copyfield tag
> >> >
> >> >success
> >> >
> >> >additional modified settings
> >> >
> >> >modified with two copyfield tags
> >> >
> >> >crash
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >Our question is, what can we do to eliminate these OOM exceptions?
> >> >
> >> >
> >>
> >
> >
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
Can you remove auto commit for bulk import. Commit at the very end?
Ahmet
On Saturday, April 5, 2014 12:16 AM, Candygram For Mongo <ca...@gmail.com> wrote:
In case the attached database.xml file didn't show up, I have pasted in the
contents below:
<dataConfig>
<dataSource
name="org_only"
type="JdbcDataSource"
driver="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
user="admin"
password="admin"
readOnly="false"
batchSize="100"
/>
<document>
<entity name="full-index" query="
select
NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
as SOLR_ID,
'ORCL.ADDRESS_ACCT_ALL'
as SOLR_CATEGORY,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
ADDRESSALLROWID,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
ADDRESSALLADDRTYPECD,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
ADDRESSALLLONGITUDE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
ADDRESSALLLATITUDE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
ADDRESSALLADDRNAME,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
ADDRESSALLCITY,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
ADDRESSALLSTATE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
ADDRESSALLEMAILADDR
from ORCL.ADDRESS_ACCT_ALL
" >
<field column="SOLR_ID" name="id" />
<field column="SOLR_CATEGORY" name="category" />
<field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
<field column="ADDRESSALLADDRTYPECD"
name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
<field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
<field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
<field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
<field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
<field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
<field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
/>
</entity>
<!-- Varaibles -->
<!-- '${dataimporter.last_index_time}' -->
</document>
</dataConfig>
On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> In this case we are indexing an Oracle database.
>
> We do not include the data-config.xml in our distribution. We store the
> database information in the database.xml file. I have attached the
> database.xml file.
>
> When we use the default merge policy settings, we get the same results.
>
>
>
> We have not tried to dump the table to a comma separated file. We think
> that dumping this size table to disk will introduce other memory problems
> with big file management. We have not tested that case.
>
>
> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Hi,
>>
>> Which database are you using? Can you send us data-config.xml?
>>
>> What happens when you use default merge policy settings?
>>
>> What happens when you dump your table to Comma Separated File and fed
>> that file to solr?
>>
>> Ahmet
>>
>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>> The ramBufferSizeMB was set to 6MB only on the test system to make the
>> system crash sooner. In production that tag is commented out which
>> I believe forces the default value to be used.
>>
>>
>>
>>
>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>
>> Hi,
>> >
>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>> >
>> >Ahmet
>> >
>> >
>> >
>> >
>> >
>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>> >
>> >*SOLR/Lucene version: *4.2.1*
>> >
>> >
>> >*JVM version:
>> >
>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>> >
>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>> >
>> >
>> >
>> >*Indexer startup command:
>> >
>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>> >
>> >
>> >
>> >java " %JVMARGS% ^
>> >
>> >-Dcom.sun.management.jmxremote.port=1092 ^
>> >
>> >-Dcom.sun.management.jmxremote.ssl=false ^
>> >
>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>> >
>> >-jar start.jar
>> >
>> >
>> >
>> >*SOLR indexing HTTP parameters request:
>> >
>> >webapp=/solr path=/dataimport
>> >params={clean=false&command=full-import&wt=javabin&version=2}
>> >
>> >
>> >
>> >We are getting a Java heap OOM exception when indexing (updating) 27
>> >million records. If we increase the Java heap memory settings the
>> problem
>> >goes away but we believe the problem has not been fixed and that we will
>> >eventually get the same OOM exception. We have other processes on the
>> >server that also require resources so we cannot continually increase the
>> >memory settings to resolve the OOM issue. We are trying to find a way to
>> >configure the SOLR instance to reduce or preferably eliminate the
>> >possibility of an OOM exception.
>> >
>> >
>> >
>> >We can reproduce the problem on a test machine. We set the Java heap
>> >memory size to 64MB to accelerate the exception. If we increase this
>> >setting the same problems occurs, just hours later. In the test
>> >environment, we are using the following parameters:
>> >
>> >
>> >
>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> >
>> >
>> >
>> >Normally we use the default solrconfig.xml file with only the following
>> jar
>> >file references added:
>> >
>> >
>> >
>> ><lib path="../../../../default/lib/common.jar" />
>> >
>> ><lib path="../../../../default/lib/webapp.jar" />
>> >
>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>> >
>> >
>> >
>> >Using these values and trying to index 6 million records from the
>> database,
>> >the Java Heap Out of Memory exception is thrown very quickly.
>> >
>> >
>> >
>> >We were able to complete a successful indexing by further modifying the
>> >solrconfig.xml and removing all or all but one <copyfield> tags from the
>> >schema.xml file.
>> >
>> >
>> >
>> >The following solrconfig.xml values were modified:
>> >
>> >
>> >
>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>> >
>> >
>> >
>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> >
>> ><int name="maxMergeAtOnce">2</int>
>> >
>> ><int name="maxMergeAtOnceExplicit">2</int>
>> >
>> ><int name="segmentsPerTier">10</int>
>> >
>> ><int name="maxMergedSegmentMB">150</int>
>> >
>> ></mergePolicy>
>> >
>> >
>> >
>> ><autoCommit>
>> >
>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>> >
>> ><openSearcher>false</openSearcher>
>> >
>> ></autoCommit>
>> >
>> >
>> >
>> >Using our customized schema.xml file with two or more <copyfield> tags,
>> the
>> >OOM exception is always thrown. Based on the errors, the problem occurs
>> >when the process was trying to do the merge. The error is provided
>> below:
>> >
>> >
>> >
>> >Exception in thread "Lucene Merge Thread #156"
>> >org.apache.lucene.index.MergePolicy$MergeException:
>> >java.lang.OutOfMemoryError: Java heap space
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> >
>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>> >
>> > at
>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>> >
>> > at
>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>> >
>> > at
>>
>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>> >
>> > at
>>
>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>> >
>> > at
>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>> >
>> > at
>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> >
>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>> >
>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>> >hit an OutOfMemoryError; cannot commit
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>> >
>> > at
>>
>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>> >
>> > at
>>
>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>> >
>> > at
>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>> >
>> > at
>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >
>> > at
>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >
>> > at
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >
>> > at
>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> >
>> > at
>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> >
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >
>> > at java.lang.Thread.run(Thread.java:722)
>> >
>> >
>> >
>> >We think but are not 100% sure that the problem is related to the merge.
>> >
>> >
>> >
>> >Normally our schema.xml contains a lot of field specifications (like the
>> >ones seen in the file fragment below):
>> >
>> >
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> dest="ADDRESS.RECORD_ID.case_abc"
>> >/>
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>> >
>> >
>> >
>> >In tests using the default file schema.xml and no <copyfield> tags,
>> >indexing completed successfully. 6 million records produced a 900 MB
>> data
>> >directory.
>> >
>> >
>> >
>> >When I included just one <copyfield> tag, indexing completed
>> successfully. 6
>> >million records produced a 990 MB data directory (90 MB bigger).
>> >
>> >
>> >
>> >When I included just two <copyfield> tags, the index crashed with an OOM
>> >exception.
>> >
>> >
>> >
>> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>> the
>> >crash.
>> >
>> >
>> >
>> >The net of our test results I as follows:
>> >
>> >
>> >
>> >*solrconfig.xml*
>> >
>> >*schema.xml*
>> >
>> >*result*
>> >
>> >
>> >default plus only jar references
>> >
>> >default (no copyfield tags)
>> >
>> >success
>> >
>> >default plus only jar references
>> >
>> >modified with one copyfield tag
>> >
>> >success
>> >
>> >default plus only jar references
>> >
>> >modified with two copyfield tags
>> >
>> >crash
>> >
>> >additional modified settings
>> >
>> >default (no copyfield tags)
>> >
>> >success
>> >
>> >additional modified settings
>> >
>> >modified with one copyfield tag
>> >
>> >success
>> >
>> >additional modified settings
>> >
>> >modified with two copyfield tags
>> >
>> >crash
>> >
>> >
>> >
>> >
>> >
>> >Our question is, what can we do to eliminate these OOM exceptions?
>> >
>> >
>>
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
In case the attached database.xml file didn't show up, I have pasted in the
contents below:
<dataConfig>
<dataSource
name="org_only"
type="JdbcDataSource"
driver="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@test2.abc.com:1521:ORCL"
user="admin"
password="admin"
readOnly="false"
batchSize="100"
/>
<document>
<entity name="full-index" query="
select
NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(100)), 'null')
as SOLR_ID,
'ORCL.ADDRESS_ACCT_ALL'
as SOLR_CATEGORY,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.RECORD_ID as varchar2(255)), ' ') as
ADDRESSALLROWID,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_TYPE_CD as varchar2(255)), ' ') as
ADDRESSALLADDRTYPECD,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.LONGITUDE as varchar2(255)), ' ') as
ADDRESSALLLONGITUDE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.LATITUDE as varchar2(255)), ' ') as
ADDRESSALLLATITUDE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.ADDR_NAME as varchar2(255)), ' ') as
ADDRESSALLADDRNAME,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.CITY as varchar2(255)), ' ') as
ADDRESSALLCITY,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.STATE as varchar2(255)), ' ') as
ADDRESSALLSTATE,
NVL(cast(ORCL.ADDRESS_ACCT_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
ADDRESSALLEMAILADDR
from ORCL.ADDRESS_ACCT_ALL
" >
<field column="SOLR_ID" name="id" />
<field column="SOLR_CATEGORY" name="category" />
<field column="ADDRESSALLROWID" name="ADDRESS_ACCT_ALL.RECORD_ID_abc" />
<field column="ADDRESSALLADDRTYPECD"
name="ADDRESS_ACCT_ALL.ADDR_TYPE_CD_abc" />
<field column="ADDRESSALLLONGITUDE" name="ADDRESS_ACCT_ALL.LONGITUDE_abc" />
<field column="ADDRESSALLLATITUDE" name="ADDRESS_ACCT_ALL.LATITUDE_abc" />
<field column="ADDRESSALLADDRNAME" name="ADDRESS_ACCT_ALL.ADDR_NAME_abc" />
<field column="ADDRESSALLCITY" name="ADDRESS_ACCT_ALL.CITY_abc" />
<field column="ADDRESSALLSTATE" name="ADDRESS_ACCT_ALL.STATE_abc" />
<field column="ADDRESSALLEMAILADDR" name="ADDRESS_ACCT_ALL.EMAIL_ADDR_abc"
/>
</entity>
<!-- Varaibles -->
<!-- '${dataimporter.last_index_time}' -->
</document>
</dataConfig>
On Fri, Apr 4, 2014 at 11:55 AM, Candygram For Mongo <
candygram.for.mongo@gmail.com> wrote:
> In this case we are indexing an Oracle database.
>
> We do not include the data-config.xml in our distribution. We store the
> database information in the database.xml file. I have attached the
> database.xml file.
>
> When we use the default merge policy settings, we get the same results.
>
>
>
> We have not tried to dump the table to a comma separated file. We think
> that dumping this size table to disk will introduce other memory problems
> with big file management. We have not tested that case.
>
>
> On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Hi,
>>
>> Which database are you using? Can you send us data-config.xml?
>>
>> What happens when you use default merge policy settings?
>>
>> What happens when you dump your table to Comma Separated File and fed
>> that file to solr?
>>
>> Ahmet
>>
>> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>>
>> The ramBufferSizeMB was set to 6MB only on the test system to make the
>> system crash sooner. In production that tag is commented out which
>> I believe forces the default value to be used.
>>
>>
>>
>>
>> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>>
>> Hi,
>> >
>> >out of curiosity, why did you set ramBufferSizeMB to 6?
>> >
>> >Ahmet
>> >
>> >
>> >
>> >
>> >
>> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
>> candygram.for.mongo@gmail.com> wrote:
>> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>> >
>> >*SOLR/Lucene version: *4.2.1*
>> >
>> >
>> >*JVM version:
>> >
>> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>> >
>> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>> >
>> >
>> >
>> >*Indexer startup command:
>> >
>> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>> >
>> >
>> >
>> >java " %JVMARGS% ^
>> >
>> >-Dcom.sun.management.jmxremote.port=1092 ^
>> >
>> >-Dcom.sun.management.jmxremote.ssl=false ^
>> >
>> >-Dcom.sun.management.jmxremote.authenticate=false ^
>> >
>> >-jar start.jar
>> >
>> >
>> >
>> >*SOLR indexing HTTP parameters request:
>> >
>> >webapp=/solr path=/dataimport
>> >params={clean=false&command=full-import&wt=javabin&version=2}
>> >
>> >
>> >
>> >We are getting a Java heap OOM exception when indexing (updating) 27
>> >million records. If we increase the Java heap memory settings the
>> problem
>> >goes away but we believe the problem has not been fixed and that we will
>> >eventually get the same OOM exception. We have other processes on the
>> >server that also require resources so we cannot continually increase the
>> >memory settings to resolve the OOM issue. We are trying to find a way to
>> >configure the SOLR instance to reduce or preferably eliminate the
>> >possibility of an OOM exception.
>> >
>> >
>> >
>> >We can reproduce the problem on a test machine. We set the Java heap
>> >memory size to 64MB to accelerate the exception. If we increase this
>> >setting the same problems occurs, just hours later. In the test
>> >environment, we are using the following parameters:
>> >
>> >
>> >
>> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>> >
>> >
>> >
>> >Normally we use the default solrconfig.xml file with only the following
>> jar
>> >file references added:
>> >
>> >
>> >
>> ><lib path="../../../../default/lib/common.jar" />
>> >
>> ><lib path="../../../../default/lib/webapp.jar" />
>> >
>> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>> >
>> >
>> >
>> >Using these values and trying to index 6 million records from the
>> database,
>> >the Java Heap Out of Memory exception is thrown very quickly.
>> >
>> >
>> >
>> >We were able to complete a successful indexing by further modifying the
>> >solrconfig.xml and removing all or all but one <copyfield> tags from the
>> >schema.xml file.
>> >
>> >
>> >
>> >The following solrconfig.xml values were modified:
>> >
>> >
>> >
>> ><ramBufferSizeMB>6</ramBufferSizeMB>
>> >
>> >
>> >
>> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> >
>> ><int name="maxMergeAtOnce">2</int>
>> >
>> ><int name="maxMergeAtOnceExplicit">2</int>
>> >
>> ><int name="segmentsPerTier">10</int>
>> >
>> ><int name="maxMergedSegmentMB">150</int>
>> >
>> ></mergePolicy>
>> >
>> >
>> >
>> ><autoCommit>
>> >
>> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>> >
>> ><openSearcher>false</openSearcher>
>> >
>> ></autoCommit>
>> >
>> >
>> >
>> >Using our customized schema.xml file with two or more <copyfield> tags,
>> the
>> >OOM exception is always thrown. Based on the errors, the problem occurs
>> >when the process was trying to do the merge. The error is provided
>> below:
>> >
>> >
>> >
>> >Exception in thread "Lucene Merge Thread #156"
>> >org.apache.lucene.index.MergePolicy$MergeException:
>> >java.lang.OutOfMemoryError: Java heap space
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> >
>> >Caused by: java.lang.OutOfMemoryError: Java heap space
>> >
>> > at
>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>> >
>> > at
>>
>> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>> >
>> > at
>>
>> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>> >
>> > at
>>
>> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>> >
>> > at
>> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>> >
>> > at
>> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> >
>> > at
>>
>> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> >
>> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>> >
>> >SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>> >hit an OutOfMemoryError; cannot commit
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>> >
>> > at
>>
>> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>> >
>> > at
>> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>> >
>> > at
>>
>> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>> >
>> > at
>> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>> >
>> > at
>> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >
>> > at
>> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >
>> > at
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >
>> > at
>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> >
>> > at
>>
>> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> >
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >
>> > at
>>
>> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >
>> > at java.lang.Thread.run(Thread.java:722)
>> >
>> >
>> >
>> >We think but are not 100% sure that the problem is related to the merge.
>> >
>> >
>> >
>> >Normally our schema.xml contains a lot of field specifications (like the
>> >ones seen in the file fragment below):
>> >
>> >
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> dest="ADDRESS.RECORD_ID.case_abc"
>> >/>
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>> >
>> ><copyField source="ADDRESS.RECORD_ID_abc"
>> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>> >
>> >
>> >
>> >In tests using the default file schema.xml and no <copyfield> tags,
>> >indexing completed successfully. 6 million records produced a 900 MB
>> data
>> >directory.
>> >
>> >
>> >
>> >When I included just one <copyfield> tag, indexing completed
>> successfully. 6
>> >million records produced a 990 MB data directory (90 MB bigger).
>> >
>> >
>> >
>> >When I included just two <copyfield> tags, the index crashed with an OOM
>> >exception.
>> >
>> >
>> >
>> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed
>> the
>> >crash.
>> >
>> >
>> >
>> >The net of our test results I as follows:
>> >
>> >
>> >
>> >*solrconfig.xml*
>> >
>> >*schema.xml*
>> >
>> >*result*
>> >
>> >
>> >default plus only jar references
>> >
>> >default (no copyfield tags)
>> >
>> >success
>> >
>> >default plus only jar references
>> >
>> >modified with one copyfield tag
>> >
>> >success
>> >
>> >default plus only jar references
>> >
>> >modified with two copyfield tags
>> >
>> >crash
>> >
>> >additional modified settings
>> >
>> >default (no copyfield tags)
>> >
>> >success
>> >
>> >additional modified settings
>> >
>> >modified with one copyfield tag
>> >
>> >success
>> >
>> >additional modified settings
>> >
>> >modified with two copyfield tags
>> >
>> >crash
>> >
>> >
>> >
>> >
>> >
>> >Our question is, what can we do to eliminate these OOM exceptions?
>> >
>> >
>>
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
In this case we are indexing an Oracle database.
We do not include the data-config.xml in our distribution. We store the
database information in the database.xml file. I have attached the
database.xml file.
When we use the default merge policy settings, we get the same results.
We have not tried to dump the table to a comma separated file. We think
that dumping this size table to disk will introduce other memory problems
with big file management. We have not tested that case.
On Fri, Apr 4, 2014 at 7:25 AM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> Which database are you using? Can you send us data-config.xml?
>
> What happens when you use default merge policy settings?
>
> What happens when you dump your table to Comma Separated File and fed that
> file to solr?
>
> Ahmet
>
> On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
>
> The ramBufferSizeMB was set to 6MB only on the test system to make the
> system crash sooner. In production that tag is commented out which
> I believe forces the default value to be used.
>
>
>
>
> On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
> Hi,
> >
> >out of curiosity, why did you set ramBufferSizeMB to 6?
> >
> >Ahmet
> >
> >
> >
> >
> >
> >On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> >*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
> >
> >*SOLR/Lucene version: *4.2.1*
> >
> >
> >*JVM version:
> >
> >Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >
> >Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >
> >
> >
> >*Indexer startup command:
> >
> >set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
> >
> >
> >
> >java " %JVMARGS% ^
> >
> >-Dcom.sun.management.jmxremote.port=1092 ^
> >
> >-Dcom.sun.management.jmxremote.ssl=false ^
> >
> >-Dcom.sun.management.jmxremote.authenticate=false ^
> >
> >-jar start.jar
> >
> >
> >
> >*SOLR indexing HTTP parameters request:
> >
> >webapp=/solr path=/dataimport
> >params={clean=false&command=full-import&wt=javabin&version=2}
> >
> >
> >
> >We are getting a Java heap OOM exception when indexing (updating) 27
> >million records. If we increase the Java heap memory settings the problem
> >goes away but we believe the problem has not been fixed and that we will
> >eventually get the same OOM exception. We have other processes on the
> >server that also require resources so we cannot continually increase the
> >memory settings to resolve the OOM issue. We are trying to find a way to
> >configure the SOLR instance to reduce or preferably eliminate the
> >possibility of an OOM exception.
> >
> >
> >
> >We can reproduce the problem on a test machine. We set the Java heap
> >memory size to 64MB to accelerate the exception. If we increase this
> >setting the same problems occurs, just hours later. In the test
> >environment, we are using the following parameters:
> >
> >
> >
> >JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
> >
> >
> >
> >Normally we use the default solrconfig.xml file with only the following
> jar
> >file references added:
> >
> >
> >
> ><lib path="../../../../default/lib/common.jar" />
> >
> ><lib path="../../../../default/lib/webapp.jar" />
> >
> ><lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >
> >
> >
> >Using these values and trying to index 6 million records from the
> database,
> >the Java Heap Out of Memory exception is thrown very quickly.
> >
> >
> >
> >We were able to complete a successful indexing by further modifying the
> >solrconfig.xml and removing all or all but one <copyfield> tags from the
> >schema.xml file.
> >
> >
> >
> >The following solrconfig.xml values were modified:
> >
> >
> >
> ><ramBufferSizeMB>6</ramBufferSizeMB>
> >
> >
> >
> ><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >
> ><int name="maxMergeAtOnce">2</int>
> >
> ><int name="maxMergeAtOnceExplicit">2</int>
> >
> ><int name="segmentsPerTier">10</int>
> >
> ><int name="maxMergedSegmentMB">150</int>
> >
> ></mergePolicy>
> >
> >
> >
> ><autoCommit>
> >
> ><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
> >
> ><openSearcher>false</openSearcher>
> >
> ></autoCommit>
> >
> >
> >
> >Using our customized schema.xml file with two or more <copyfield> tags,
> the
> >OOM exception is always thrown. Based on the errors, the problem occurs
> >when the process was trying to do the merge. The error is provided below:
> >
> >
> >
> >Exception in thread "Lucene Merge Thread #156"
> >org.apache.lucene.index.MergePolicy$MergeException:
> >java.lang.OutOfMemoryError: Java heap space
> >
> > at
>
> >org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> >
> > at
>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> >
> >Caused by: java.lang.OutOfMemoryError: Java heap space
> >
> > at
>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
> >
> > at
>
> >org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
> >
> > at
>
> >org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
> >
> > at
>
> >org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
> >
> > at
> >org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
> >
> > at
> >org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
> >
> > at
> >org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> >
> > at
> >org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> >
> > at
>
> >org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> >
> > at
>
> >org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> >
> >Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
> >
> >SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
> >hit an OutOfMemoryError; cannot commit
> >
> > at
> >org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
> >
> > at
>
> >org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
> >
> > at
> >org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
> >
> > at
> >org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
> >
> > at
>
> >org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
> >
> > at
> >org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> >
> > at
> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >
> > at
> >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >
> > at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >
> > at
>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >
> > at
>
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >
> > at
>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> > at
>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >
> > at java.lang.Thread.run(Thread.java:722)
> >
> >
> >
> >We think but are not 100% sure that the problem is related to the merge.
> >
> >
> >
> >Normally our schema.xml contains a lot of field specifications (like the
> >ones seen in the file fragment below):
> >
> >
> >
> ><copyField source="ADDRESS.RECORD_ID_abc"
> dest="ADDRESS.RECORD_ID.case_abc"
> >/>
> >
> ><copyField source="ADDRESS.RECORD_ID_abc"
> >dest="ADDRESS.RECORD_ID.case.soundex_abc" />
> >
> ><copyField source="ADDRESS.RECORD_ID_abc"
> >dest="ADDRESS.RECORD_ID.case_nvl_abc" />
> >
> >
> >
> >In tests using the default file schema.xml and no <copyfield> tags,
> >indexing completed successfully. 6 million records produced a 900 MB data
> >directory.
> >
> >
> >
> >When I included just one <copyfield> tag, indexing completed
> successfully. 6
> >million records produced a 990 MB data directory (90 MB bigger).
> >
> >
> >
> >When I included just two <copyfield> tags, the index crashed with an OOM
> >exception.
> >
> >
> >
> >Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
> >crash.
> >
> >
> >
> >The net of our test results I as follows:
> >
> >
> >
> >*solrconfig.xml*
> >
> >*schema.xml*
> >
> >*result*
> >
> >
> >default plus only jar references
> >
> >default (no copyfield tags)
> >
> >success
> >
> >default plus only jar references
> >
> >modified with one copyfield tag
> >
> >success
> >
> >default plus only jar references
> >
> >modified with two copyfield tags
> >
> >crash
> >
> >additional modified settings
> >
> >default (no copyfield tags)
> >
> >success
> >
> >additional modified settings
> >
> >modified with one copyfield tag
> >
> >success
> >
> >additional modified settings
> >
> >modified with two copyfield tags
> >
> >crash
> >
> >
> >
> >
> >
> >Our question is, what can we do to eliminate these OOM exceptions?
> >
> >
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
Which database are you using? Can you send us data-config.xml?
What happens when you use default merge policy settings?
What happens when you dump your table to Comma Separated File and fed that file to solr?
Ahmet
On Friday, April 4, 2014 5:10 PM, Candygram For Mongo <ca...@gmail.com> wrote:
The ramBufferSizeMB was set to 6MB only on the test system to make the system crash sooner. In production that tag is commented out which I believe forces the default value to be used.
On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
Hi,
>
>out of curiosity, why did you set ramBufferSizeMB to 6?
>
>Ahmet
>
>
>
>
>
>On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <ca...@gmail.com> wrote:
>*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>
>*SOLR/Lucene version: *4.2.1*
>
>
>*JVM version:
>
>Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>
>Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>
>
>
>*Indexer startup command:
>
>set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>
>
>
>java " %JVMARGS% ^
>
>-Dcom.sun.management.jmxremote.port=1092 ^
>
>-Dcom.sun.management.jmxremote.ssl=false ^
>
>-Dcom.sun.management.jmxremote.authenticate=false ^
>
>-jar start.jar
>
>
>
>*SOLR indexing HTTP parameters request:
>
>webapp=/solr path=/dataimport
>params={clean=false&command=full-import&wt=javabin&version=2}
>
>
>
>We are getting a Java heap OOM exception when indexing (updating) 27
>million records. If we increase the Java heap memory settings the problem
>goes away but we believe the problem has not been fixed and that we will
>eventually get the same OOM exception. We have other processes on the
>server that also require resources so we cannot continually increase the
>memory settings to resolve the OOM issue. We are trying to find a way to
>configure the SOLR instance to reduce or preferably eliminate the
>possibility of an OOM exception.
>
>
>
>We can reproduce the problem on a test machine. We set the Java heap
>memory size to 64MB to accelerate the exception. If we increase this
>setting the same problems occurs, just hours later. In the test
>environment, we are using the following parameters:
>
>
>
>JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>
>
>
>Normally we use the default solrconfig.xml file with only the following jar
>file references added:
>
>
>
><lib path="../../../../default/lib/common.jar" />
>
><lib path="../../../../default/lib/webapp.jar" />
>
><lib path="../../../../default/lib/commons-pool-1.4.jar" />
>
>
>
>Using these values and trying to index 6 million records from the database,
>the Java Heap Out of Memory exception is thrown very quickly.
>
>
>
>We were able to complete a successful indexing by further modifying the
>solrconfig.xml and removing all or all but one <copyfield> tags from the
>schema.xml file.
>
>
>
>The following solrconfig.xml values were modified:
>
>
>
><ramBufferSizeMB>6</ramBufferSizeMB>
>
>
>
><mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>
><int name="maxMergeAtOnce">2</int>
>
><int name="maxMergeAtOnceExplicit">2</int>
>
><int name="segmentsPerTier">10</int>
>
><int name="maxMergedSegmentMB">150</int>
>
></mergePolicy>
>
>
>
><autoCommit>
>
><maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>
><openSearcher>false</openSearcher>
>
></autoCommit>
>
>
>
>Using our customized schema.xml file with two or more <copyfield> tags, the
>OOM exception is always thrown. Based on the errors, the problem occurs
>when the process was trying to do the merge. The error is provided below:
>
>
>
>Exception in thread "Lucene Merge Thread #156"
>org.apache.lucene.index.MergePolicy$MergeException:
>java.lang.OutOfMemoryError: Java heap space
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>
>Caused by: java.lang.OutOfMemoryError: Java heap space
>
> at
>org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>
> at
>org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>
> at
>org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>
> at
>org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>
> at
>org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>
> at
>org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>
> at
>org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>
> at
>org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>
> at
>org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>
>Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>
>SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
>hit an OutOfMemoryError; cannot commit
>
> at
>org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>
> at
>org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>
> at
>org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>
> at
>org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>
> at
>org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>
> at
>org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>
> at
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:722)
>
>
>
>We think but are not 100% sure that the problem is related to the merge.
>
>
>
>Normally our schema.xml contains a lot of field specifications (like the
>ones seen in the file fragment below):
>
>
>
><copyField source="ADDRESS.RECORD_ID_abc" dest="ADDRESS.RECORD_ID.case_abc"
>/>
>
><copyField source="ADDRESS.RECORD_ID_abc"
>dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>
><copyField source="ADDRESS.RECORD_ID_abc"
>dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>
>
>
>In tests using the default file schema.xml and no <copyfield> tags,
>indexing completed successfully. 6 million records produced a 900 MB data
>directory.
>
>
>
>When I included just one <copyfield> tag, indexing completed successfully. 6
>million records produced a 990 MB data directory (90 MB bigger).
>
>
>
>When I included just two <copyfield> tags, the index crashed with an OOM
>exception.
>
>
>
>Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
>crash.
>
>
>
>The net of our test results I as follows:
>
>
>
>*solrconfig.xml*
>
>*schema.xml*
>
>*result*
>
>
>default plus only jar references
>
>default (no copyfield tags)
>
>success
>
>default plus only jar references
>
>modified with one copyfield tag
>
>success
>
>default plus only jar references
>
>modified with two copyfield tags
>
>crash
>
>additional modified settings
>
>default (no copyfield tags)
>
>success
>
>additional modified settings
>
>modified with one copyfield tag
>
>success
>
>additional modified settings
>
>modified with two copyfield tags
>
>crash
>
>
>
>
>
>Our question is, what can we do to eliminate these OOM exceptions?
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Candygram For Mongo <ca...@gmail.com>.
The ramBufferSizeMB was set to 6MB only on the test system to make the
system crash sooner. In production that tag is commented out which
I believe forces the default value to be used.
On Thu, Apr 3, 2014 at 5:46 PM, Ahmet Arslan <io...@yahoo.com> wrote:
> Hi,
>
> out of curiosity, why did you set ramBufferSizeMB to 6?
>
> Ahmet
>
>
>
>
> On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <
> candygram.for.mongo@gmail.com> wrote:
> *Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
>
> *SOLR/Lucene version: *4.2.1*
>
> *JVM version:
>
> Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
>
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>
>
>
> *Indexer startup command:
>
> set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
>
>
>
> java " %JVMARGS% ^
>
> -Dcom.sun.management.jmxremote.port=1092 ^
>
> -Dcom.sun.management.jmxremote.ssl=false ^
>
> -Dcom.sun.management.jmxremote.authenticate=false ^
>
> -jar start.jar
>
>
>
> *SOLR indexing HTTP parameters request:
>
> webapp=/solr path=/dataimport
> params={clean=false&command=full-import&wt=javabin&version=2}
>
>
>
> We are getting a Java heap OOM exception when indexing (updating) 27
> million records. If we increase the Java heap memory settings the problem
> goes away but we believe the problem has not been fixed and that we will
> eventually get the same OOM exception. We have other processes on the
> server that also require resources so we cannot continually increase the
> memory settings to resolve the OOM issue. We are trying to find a way to
> configure the SOLR instance to reduce or preferably eliminate the
> possibility of an OOM exception.
>
>
>
> We can reproduce the problem on a test machine. We set the Java heap
> memory size to 64MB to accelerate the exception. If we increase this
> setting the same problems occurs, just hours later. In the test
> environment, we are using the following parameters:
>
>
>
> JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
>
>
>
> Normally we use the default solrconfig.xml file with only the following jar
> file references added:
>
>
>
> <lib path="../../../../default/lib/common.jar" />
>
> <lib path="../../../../default/lib/webapp.jar" />
>
> <lib path="../../../../default/lib/commons-pool-1.4.jar" />
>
>
>
> Using these values and trying to index 6 million records from the database,
> the Java Heap Out of Memory exception is thrown very quickly.
>
>
>
> We were able to complete a successful indexing by further modifying the
> solrconfig.xml and removing all or all but one <copyfield> tags from the
> schema.xml file.
>
>
>
> The following solrconfig.xml values were modified:
>
>
>
> <ramBufferSizeMB>6</ramBufferSizeMB>
>
>
>
> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>
> <int name="maxMergeAtOnce">2</int>
>
> <int name="maxMergeAtOnceExplicit">2</int>
>
> <int name="segmentsPerTier">10</int>
>
> <int name="maxMergedSegmentMB">150</int>
>
> </mergePolicy>
>
>
>
> <autoCommit>
>
> <maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
>
> <openSearcher>false</openSearcher>
>
> </autoCommit>
>
>
>
> Using our customized schema.xml file with two or more <copyfield> tags, the
> OOM exception is always thrown. Based on the errors, the problem occurs
> when the process was trying to do the merge. The error is provided below:
>
>
>
> Exception in thread "Lucene Merge Thread #156"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
>
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> at
>
> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
>
> at
>
> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
>
> at
>
> org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
>
> at
> org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
>
> at
> org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
>
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
>
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
>
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
>
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>
> Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
>
> SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
> hit an OutOfMemoryError; cannot commit
>
> at
> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
>
> at
>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
>
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
>
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
>
> at
>
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
>
> at
> org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:722)
>
>
>
> We think but are not 100% sure that the problem is related to the merge.
>
>
>
> Normally our schema.xml contains a lot of field specifications (like the
> ones seen in the file fragment below):
>
>
>
> <copyField source="ADDRESS.RECORD_ID_abc" dest="ADDRESS.RECORD_ID.case_abc"
> />
>
> <copyField source="ADDRESS.RECORD_ID_abc"
> dest="ADDRESS.RECORD_ID.case.soundex_abc" />
>
> <copyField source="ADDRESS.RECORD_ID_abc"
> dest="ADDRESS.RECORD_ID.case_nvl_abc" />
>
>
>
> In tests using the default file schema.xml and no <copyfield> tags,
> indexing completed successfully. 6 million records produced a 900 MB data
> directory.
>
>
>
> When I included just one <copyfield> tag, indexing completed
> successfully. 6
> million records produced a 990 MB data directory (90 MB bigger).
>
>
>
> When I included just two <copyfield> tags, the index crashed with an OOM
> exception.
>
>
>
> Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
> crash.
>
>
>
> The net of our test results I as follows:
>
>
>
> *solrconfig.xml*
>
> *schema.xml*
>
> *result*
>
> default plus only jar references
>
> default (no copyfield tags)
>
> success
>
> default plus only jar references
>
> modified with one copyfield tag
>
> success
>
> default plus only jar references
>
> modified with two copyfield tags
>
> crash
>
> additional modified settings
>
> default (no copyfield tags)
>
> success
>
> additional modified settings
>
> modified with one copyfield tag
>
> success
>
> additional modified settings
>
> modified with two copyfield tags
>
> crash
>
>
>
>
>
> Our question is, what can we do to eliminate these OOM exceptions?
>
>
Re: Full Indexing is Causing a Java Heap Out of Memory Exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi,
out of curiosity, why did you set ramBufferSizeMB to 6?
Ahmet
On Friday, April 4, 2014 3:27 AM, Candygram For Mongo <ca...@gmail.com> wrote:
*Main issue: Full Indexing is Causing a Java Heap Out of Memory Exception
*SOLR/Lucene version: *4.2.1*
*JVM version:
Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
*Indexer startup command:
set JVMARGS=-XX:MaxPermSize=364m -Xss256K -Xmx6144m -Xms6144m
java " %JVMARGS% ^
-Dcom.sun.management.jmxremote.port=1092 ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Dcom.sun.management.jmxremote.authenticate=false ^
-jar start.jar
*SOLR indexing HTTP parameters request:
webapp=/solr path=/dataimport
params={clean=false&command=full-import&wt=javabin&version=2}
We are getting a Java heap OOM exception when indexing (updating) 27
million records. If we increase the Java heap memory settings the problem
goes away but we believe the problem has not been fixed and that we will
eventually get the same OOM exception. We have other processes on the
server that also require resources so we cannot continually increase the
memory settings to resolve the OOM issue. We are trying to find a way to
configure the SOLR instance to reduce or preferably eliminate the
possibility of an OOM exception.
We can reproduce the problem on a test machine. We set the Java heap
memory size to 64MB to accelerate the exception. If we increase this
setting the same problems occurs, just hours later. In the test
environment, we are using the following parameters:
JVMARGS=-XX:MaxPermSize=64m -Xss256K -Xmx64m -Xms64m
Normally we use the default solrconfig.xml file with only the following jar
file references added:
<lib path="../../../../default/lib/common.jar" />
<lib path="../../../../default/lib/webapp.jar" />
<lib path="../../../../default/lib/commons-pool-1.4.jar" />
Using these values and trying to index 6 million records from the database,
the Java Heap Out of Memory exception is thrown very quickly.
We were able to complete a successful indexing by further modifying the
solrconfig.xml and removing all or all but one <copyfield> tags from the
schema.xml file.
The following solrconfig.xml values were modified:
<ramBufferSizeMB>6</ramBufferSizeMB>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">2</int>
<int name="maxMergeAtOnceExplicit">2</int>
<int name="segmentsPerTier">10</int>
<int name="maxMergedSegmentMB">150</int>
</mergePolicy>
<autoCommit>
<maxDocs>15000</maxDocs> <!-- This tag was maxTime, before this -- >
<openSearcher>false</openSearcher>
</autoCommit>
Using our customized schema.xml file with two or more <copyfield> tags, the
OOM exception is always thrown. Based on the errors, the problem occurs
when the process was trying to do the merge. The error is provided below:
Exception in thread "Lucene Merge Thread #156"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
at
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
at
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
at
org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:233)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
Mar 12, 2014 12:17:40 AM org.apache.solr.common.SolrException log
SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
hit an OutOfMemoryError; cannot commit
at
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:3971)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2744)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:536)
at
org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
We think but are not 100% sure that the problem is related to the merge.
Normally our schema.xml contains a lot of field specifications (like the
ones seen in the file fragment below):
<copyField source="ADDRESS.RECORD_ID_abc" dest="ADDRESS.RECORD_ID.case_abc"
/>
<copyField source="ADDRESS.RECORD_ID_abc"
dest="ADDRESS.RECORD_ID.case.soundex_abc" />
<copyField source="ADDRESS.RECORD_ID_abc"
dest="ADDRESS.RECORD_ID.case_nvl_abc" />
In tests using the default file schema.xml and no <copyfield> tags,
indexing completed successfully. 6 million records produced a 900 MB data
directory.
When I included just one <copyfield> tag, indexing completed successfully. 6
million records produced a 990 MB data directory (90 MB bigger).
When I included just two <copyfield> tags, the index crashed with an OOM
exception.
Changing parameters like maxMergedSegmentMB or maxDocs, only postponed the
crash.
The net of our test results I as follows:
*solrconfig.xml*
*schema.xml*
*result*
default plus only jar references
default (no copyfield tags)
success
default plus only jar references
modified with one copyfield tag
success
default plus only jar references
modified with two copyfield tags
crash
additional modified settings
default (no copyfield tags)
success
additional modified settings
modified with one copyfield tag
success
additional modified settings
modified with two copyfield tags
crash
Our question is, what can we do to eliminate these OOM exceptions?