You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ma...@apache.org on 2012/01/18 23:28:20 UTC
svn commit: r1233096 [2/13] - in /lucene/dev/branches/solrcloud: ./
dev-tools/eclipse/ dev-tools/idea/.idea/ dev-tools/idea/lucene/contrib/
dev-tools/idea/modules/analysis/kuromoji/
dev-tools/idea/solr/contrib/analysis-extras/ dev-tools/maven/modules/a...
Modified: lucene/dev/branches/solrcloud/lucene/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/CHANGES.txt?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/CHANGES.txt (original)
+++ lucene/dev/branches/solrcloud/lucene/CHANGES.txt Wed Jan 18 22:28:07 2012
@@ -226,6 +226,14 @@ Changes in backwards compatibility polic
* LUCENE-3640: Removed IndexSearcher.close(), because IndexSearcher no longer
takes a Directory and no longer "manages" IndexReaders, it is a no-op.
(Robert Muir)
+
+* LUCENE-3684: Add offsets into DocsAndPositionsEnum, and a few
+ FieldInfo.IndexOption: DOCS_AND_POSITIONS_AND_OFFSETS. (Robert
+ Muir, Mike McCandless)
+
+* LUCENE-3646: FieldCacheImpl now throws UOE on non-atomic IndexReaders. If
+ you really want a top-level fieldcache, use SlowMultiReaderWrapper.
+ (Robert Muir)
Changes in Runtime Behavior
@@ -475,11 +483,6 @@ New features
QueryParser subclasses that overrode getRangeQuery will need to be changed
to use the new getRangeQuery method. (Andrew Schurman, Mark Miller, yonik)
-* LUCENE-995: The QueryParser now interprets * as an open end for range
- queries. Literal asterisks may be represented by quoting or escaping
- (i.e. \* or "*") Custom QueryParser subclasses overriding getRangeQuery()
- will be passed null for any open endpoint. (Adriano Crestani, yonik)
-
* LUCENE-2742: Add native per-field postings format support. Codec lets you now
register a postings format for each field and which is in turn recorded
into the index. Postings formtas are maintained on a per-segment basis and be
@@ -623,6 +626,11 @@ New features
* LUCENE-3628: Norms are represented as DocValues. IndexReader exposes
a #normValues(String) method to obtain norms per field. (Simon Willnauer)
+* LUCENE-3687: Similarity#computeNorm(FieldInvertState, Norm) allows to compute
+ norm values or arbitrary precision. Instead of returning a fixed single byte
+ value, custom similarities can now set a integer, float or byte value to the
+ given Norm object. (Simon Willnauer)
+
Optimizations
* LUCENE-2588: Don't store unnecessary suffixes when writing the terms
@@ -714,6 +722,25 @@ Changes in backwards compatibility polic
contrib/queryparser. If you have used those classes in your code
just add the lucene-queryparser.jar file to your classpath.
(Uwe Schindler)
+
+* LUCENE-3681: FST now stores labels for BYTE2 input type as 2 bytes
+ instead of vInt; this can make FSTs smaller and faster, but it is a
+ break in the binary format so if you had built and saved any FSTs
+ then you need to rebuild them. (Robert Muir, Mike McCandless)
+
+* LUCENE-3679: The expert IndexReader.getFieldNames(FieldOption) API
+ has been removed and replaced with the experimental getFieldInfos
+ API. All IndexReader subclasses must implement getFieldInfos.
+ (Mike McCandless)
+
+* LUCENE-3695: Move confusing add(X) methods out of FST.Builder into
+ FST.Util. (Robert Muir, Mike McCandless)
+
+* LUCENE-3701: Added an additional argument to the expert FST.Builder
+ ctor to take FreezeTail, which you can use to (very-expertly) customize
+ the FST construction process. Pass null if you want the default
+ behavior. Added seekExact() to FSTEnum, and added FST.save/read
+ from a File. (Mike McCandless, Dawid Weiss, Robert Muir)
Security fixes
@@ -752,6 +779,12 @@ New Features
* LUCENE-3634: IndexReader's static main method was moved to a new
tool, CompoundFileExtractor, in contrib/misc. (Robert Muir, Mike
McCandless)
+
+* LUCENE-995: The QueryParser now interprets * as an open end for range
+ queries. Literal asterisks may be represented by quoting or escaping
+ (i.e. \* or "*") Custom QueryParser subclasses overriding getRangeQuery()
+ will be passed null for any open endpoint. (Ingo Renner, Adriano
+ Crestani, yonik, Mike McCandless
Bug fixes
Modified: lucene/dev/branches/solrcloud/lucene/MIGRATE.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/MIGRATE.txt?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/MIGRATE.txt (original)
+++ lucene/dev/branches/solrcloud/lucene/MIGRATE.txt Wed Jan 18 22:28:07 2012
@@ -422,13 +422,13 @@ LUCENE-1458, LUCENE-2111: Flexible Index
-* LUCENE-2308: Separate IndexableFieldType from Field instances
+* LUCENE-2308,LUCENE-3453: Separate IndexableFieldType from Field instances
With this change, the indexing details (indexed, tokenized, norms,
indexOptions, stored, etc.) are moved into a separate FieldType
instance (rather than being stored directly on the Field).
-This means you can create the IndexableFieldType instance once, up front,
+This means you can create the FieldType instance once, up front,
for a given field, and then re-use that instance whenever you instantiate
the Field.
@@ -439,15 +439,21 @@ Certain field types are pre-defined sinc
IDS (does not index term frequency nor positions). This field
does not store its value, but exposes TYPE_STORED as well.
- * BinaryField: a byte[] value that's only stored.
-
* TextField: indexes and tokenizes a String, Reader or TokenStream
value, without term vectors. This field does not store its value,
but exposes TYPE_STORED as well.
+ * StoredField: field that stores its value
+
+ * DocValuesField: indexes the value as a DocValues field
+
+ * NumericField: indexes the numeric value so that NumericRangeQuery
+ can be used at search-time.
+
If your usage fits one of those common cases you can simply
-instantiate the above class. To use the TYPE_STORED variant, do this
-instead:
+instantiate the above class. If you need to store the value, you can
+add a separate StoredField to the document, or you can use
+TYPE_STORED for the field:
Field f = new Field("field", "value", StringField.TYPE_STORED);
@@ -465,9 +471,14 @@ You can of course also create your own F
t.setStored(true);
t.setOmitNorms(true);
t.setIndexOptions(IndexOptions.DOCS_AND_FREQS);
+ t.freeze();
FieldType has a freeze() method to prevent further changes.
+There is also a deprecated transition API, providing the same Index,
+Store, TermVector enums from 3.x, and Field constructors taking these
+enums.
+
When migrating from the 3.x API, if you did this before:
new Field("field", "value", Field.Store.NO, Field.Indexed.NOT_ANALYZED_NO_NORMS)
@@ -528,7 +539,7 @@ If you did this before (bytes is a byte[
you can now do this:
- new BinaryField("field", bytes)
+ new StoredField("field", bytes)
* LUCENE-3396: Analyzer.tokenStream() and .reusableTokenStream() have been made final.
It is now necessary to use Analyzer.TokenStreamComponents to define an analysis process.
@@ -539,3 +550,7 @@ you can now do this:
IndexReader.openIfChanged (a static method), and now returns null
(instead of the old reader) if there are no changes to the index, to
prevent the common pitfall of accidentally closing the old reader.
+
+* LUCENE-3687: Similarity#computeNorm() now expects a Norm object to set the computed
+ norm value instead of returning a fixed single byte value. Custom similarities can now
+ set integer, float and byte values if a single byte is not sufficient.
Modified: lucene/dev/branches/solrcloud/lucene/common-build.xml
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/common-build.xml?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/common-build.xml (original)
+++ lucene/dev/branches/solrcloud/lucene/common-build.xml Wed Jan 18 22:28:07 2012
@@ -111,6 +111,7 @@
<!-- clover wants to run with -lib, otherwise we prefer a repeatable
classpath -->
<property name="javac.includeAntRuntime" value="${run.clover}"/>
+ <property name="javac.args" value="-Xlint -Xlint:-deprecation -Xlint:-serial"/>
<property name="javadoc.link" value="http://download.oracle.com/javase/6/docs/api/"/>
<property name="javadoc.link.junit" value="http://junit.sourceforge.net/javadoc/"/>
@@ -819,7 +820,7 @@
<!-- <compilerarg line="-Xmaxwarns 10000000"/>
<compilerarg line="-Xmaxerrs 10000000"/> -->
<!-- for generics in Java 1.5: -->
- <compilerarg line="-Xlint -Xlint:-deprecation -Xlint:-serial"/>
+ <compilerarg line="${javac.args}"/>
</javac>
</sequential>
</macrodef>
Modified: lucene/dev/branches/solrcloud/lucene/contrib/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/CHANGES.txt?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/CHANGES.txt (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/CHANGES.txt Wed Jan 18 22:28:07 2012
@@ -57,7 +57,11 @@ New Features
* LUCENE-3527: Add LuceneLevenshteinDistance, which computes string distance in a compatible
way as DirectSpellChecker. This can be used to merge top-N results from more than one
SpellChecker. (James Dyer via Robert Muir)
-
+
+ * LUCENE-3496: Support grouping by DocValues. (Martijn van Groningen)
+
+ * LUCENE-3602: Added query time joining under the join module. (Martijn van Groningen, Michael McCandless)
+
API Changes
* LUCENE-2606: Changed RegexCapabilities interface to fix thread
@@ -75,17 +79,25 @@ API Changes
* LUCENE-3308: DuplicateFilter keepMode and processingMode have been converted to
enums DuplicateFilter.KeepMode and DuplicateFilter.ProcessingMode repsectively.
+
+ * LUCENE-3483: Move Function grouping collectors from Solr to grouping module.
+ (Martijn van Groningen)
-* LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
- setNorm() was deprecated. Furthermore, this class is broken, as it does
- not take position overlaps into account while recalculating norms.
- (Uwe Schindler, Robert Muir)
+ * LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
+ setNorm() was deprecated. Furthermore, this class is broken, as it does
+ not take position overlaps into account while recalculating norms.
+ (Uwe Schindler, Robert Muir)
Bug Fixes
* LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
not lowercasing the key before checking for the tag (Adriano Crestani)
+Optimizations
+
+* LUCENE-3468: Replaced last() and remove() with pollLast() in
+ FirstPassGroupingCollector (Martijn van Groningen)
+
======================= Lucene 3.6.0 ================
Changes in backwards compatibility policy
@@ -93,39 +105,68 @@ Changes in backwards compatibility polic
* LUCENE-3626: The internal implementation classes in PKIndexSplitter
and MultiPassIndexSplitter were made private as they now work
per segment. (Uwe Schindler)
+
+Changes in Runtime Behavior
-New Features
+ * LUCENE-3698: FastVectorHighlighter no longer adds a multi value separator
+ to the end of the highlighted text. (Shay Banon via Koji Sekiguchi)
-* LUCENE-3596: DirectoryTaxonomyWriter extensions can override createIndexWriterConfig()
- and modify how its internal index writer is opened. (Doron Cohen)
+New Features
-* SOLR-2982: Added phonetic encoders to contrib/analyzers/phonetic:
- Metaphone, Soundex, Caverphone, Beider-Morse, etc. (Robert Muir)
+ * LUCENE-3596: DirectoryTaxonomyWriter extensions can override createIndexWriterConfig()
+ and modify how its internal index writer is opened. (Doron Cohen)
-* LUCENE-2906: Added CJKBigramFilter that forms bigrams from StandardTokenizer or
- ICUTokenizer CJK tokens, and CJKWidthFilter that normalizes halfwidth/fullwidth.
- This filter supports unicode supplementary characters and you can toggle whether
- bigrams are formed for each of Han/Hiragana/Katakana/Hangul independently. Deprecates
- CJKTokenizer. (Tom Burton-West, Robert Muir)
+ * SOLR-2982: Added phonetic encoders to contrib/analyzers/phonetic:
+ Metaphone, Soundex, Caverphone, Beider-Morse, etc. (Robert Muir)
-* LUCENE-3634: IndexReader's static main method was moved to a new
- tool, CompoundFileExtractor, in contrib/misc. (Mike McCandless)
+ * LUCENE-2906: Added CJKBigramFilter that forms bigrams from StandardTokenizer or
+ ICUTokenizer CJK tokens, and CJKWidthFilter that normalizes halfwidth/fullwidth.
+ This filter supports unicode supplementary characters and you can toggle whether
+ bigrams are formed for each of Han/Hiragana/Katakana/Hangul independently. Deprecates
+ CJKTokenizer. (Tom Burton-West, Robert Muir)
+
+ * LUCENE-3634: IndexReader's static main method was moved to a new
+ tool, CompoundFileExtractor, in contrib/misc. (Mike McCandless)
+
+ * SOLR-3020: Add KeywordAttribute support to HunspellStemFilter. Terms marked as
+ keywords are not modified by the stemmer. (Simon Willnauer, Helge Jenssen)
+
+ * LUCENE-3305: Added Kuromoji morphological analyzer for Japanese.
+ (Christian Moen, Masaru Hasegawa, Simon Willnauer, Uwe Schindler, Mike McCandless, Robert Muir)
+
+ * LUCENE-3685: Add ToChildBlockJoinQuery and renamed previous
+ BlockJoinQuery to ToParentBlockJoinQuery, so that you can now do
+ joins in both parent to child and child to parent directions.
+ (Mike McCandless)
API Changes
-* LUCENE-3596: DirectoryTaxonomyWriter.openIndexWriter() now takes an
- openIndexWriter parameter rather than just an open-mode. (Doron Cohen)
-
-* LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
- setNorm() was deprecated. Furthermore, this class is broken, as it does
- not take position overlaps into account while recalculating norms.
- (Uwe Schindler, Robert Muir)
+ * LUCENE-3596: DirectoryTaxonomyWriter.openIndexWriter() now takes an
+ openIndexWriter parameter rather than just an open-mode. (Doron Cohen)
-Changes in runtime behavior:
+ * LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
+ setNorm() was deprecated. Furthermore, this class is broken, as it does
+ not take position overlaps into account while recalculating norms.
+ (Uwe Schindler, Robert Muir)
+
+ * LUCENE-3305: Added SegmentingTokenizerBase, which breaks text into sentences
+ with BreakIterator and allows subclasses to decompose sentences into words, or
+ use the sentence boundary information for other reasons (e.g. attribute/position increment)
+ (Robert Muir)
+
+Changes in runtime behavior
* LUCENE-3626: PKIndexSplitter and MultiPassIndexSplitter now work
per segment. (Uwe Schindler)
+Optimizations
+
+* SOLR-2888: FSTSuggester refactoring: internal storage is now UTF-8,
+ external sorting (on disk) prevents OOMs even with large data sets
+ (the bottleneck is now FST construction), code cleanups and API cleanups.
+ You should use FSTCompletionLookup (the old FSTLookup impl is deprecated).
+ (Dawid Weiss, Robert Muir)
+
Bug Fixes
* LUCENE-3600: BlockJoinQuery now supports parent docs that have no
@@ -144,10 +185,16 @@ Bug Fixes
This way if the synonym token is used for highlighting, it will
cover all tokens it had matched. (Koji Sekiguchi, Robert Muir,
Mike McCandless)
-
+
+ * LUCENE-3686: CategoryEnhancement must override Object.equals(Object).
+ (Sivan Yogev via Shai Erera)
+
+ * LUCENE-3697: SimpleBoundaryScanner does not work well when highlighting
+ at the beginning of the text. (Shay Banon via Koji Sekiguchi)
+
Documentation
- * LUCENE-3599: Javadocs for DistanceUtils.haversine() were incorrectly
+ * LUCENE-3599: Javadocs for DistanceUtils.haversine() were incorrectly
stating the expected order of the arguments (David Smiley via hossman)
======================= Lucene 3.5.0 ================
Modified: lucene/dev/branches/solrcloud/lucene/contrib/contrib-build.xml
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/contrib-build.xml?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/contrib-build.xml (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/contrib-build.xml Wed Jan 18 22:28:07 2012
@@ -196,6 +196,17 @@
<property name="analyzers-stempel.uptodate" value="true"/>
</target>
+ <property name="analyzers-kuromoji.jar" value="${common.dir}/../modules/analysis/build/kuromoji/lucene-analyzers-kuromoji-${version}.jar"/>
+ <target name="check-analyzers-kuromoji-uptodate" unless="analyzers-kuromoji.uptodate">
+ <module-uptodate name="analysis/kuromoji" jarfile="${analyzers-kuromoji.jar}" property="analyzers-kuromoji.uptodate"/>
+ </target>
+ <target name="jar-analyzers-kuromoji" unless="analyzers-kuromoji.uptodate" depends="check-analyzers-kuromoji-uptodate">
+ <ant dir="${common.dir}/../modules/analysis/kuromoji" target="jar-core" inheritAll="false">
+ <propertyset refid="uptodate.and.compiled.properties"/>
+ </ant>
+ <property name="analyzers-kuromoji.uptodate" value="true"/>
+ </target>
+
<property name="grouping.jar" value="${common.dir}/../modules/grouping/build/lucene-grouping-${version}.jar"/>
<target name="check-grouping-uptodate" unless="grouping.uptodate">
<module-uptodate name="grouping" jarfile="${grouping.jar}" property="grouping.uptodate"/>
Modified: lucene/dev/branches/solrcloud/lucene/contrib/demo/src/java/org/apache/lucene/demo/IndexFiles.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/demo/src/java/org/apache/lucene/demo/IndexFiles.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/demo/src/java/org/apache/lucene/demo/IndexFiles.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/demo/src/java/org/apache/lucene/demo/IndexFiles.java Wed Jan 18 22:28:07 2012
@@ -184,9 +184,7 @@ public class IndexFiles {
// year/month/day/hour/minutes/seconds, down the resolution you require.
// For example the long value 2011021714 would mean
// February 17, 2011, 2-3 PM.
- NumericField modifiedField = new NumericField("modified");
- modifiedField.setLongValue(file.lastModified());
- doc.add(modifiedField);
+ doc.add(new NumericField("modified", file.lastModified()));
// Add the contents of the file to a field named "contents". Specify a Reader,
// so that the text of the file is tokenized and indexed, but not stored.
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenSources.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenSources.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenSources.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenSources.java Wed Jan 18 22:28:07 2012
@@ -126,7 +126,7 @@ public class TokenSources {
private static boolean hasPositions(Terms vector) throws IOException {
final TermsEnum termsEnum = vector.iterator(null);
if (termsEnum.next() != null) {
- DocsAndPositionsEnum dpEnum = termsEnum.docsAndPositions(null, null);
+ DocsAndPositionsEnum dpEnum = termsEnum.docsAndPositions(null, null, false);
if (dpEnum != null) {
int pos = dpEnum.nextPosition();
if (pos >= 0) {
@@ -219,22 +219,21 @@ public class TokenSources {
DocsAndPositionsEnum dpEnum = null;
while ((text = termsEnum.next()) != null) {
- dpEnum = termsEnum.docsAndPositions(null, dpEnum);
- if (dpEnum == null || (!dpEnum.attributes().hasAttribute(OffsetAttribute.class))) {
+ dpEnum = termsEnum.docsAndPositions(null, dpEnum, true);
+ if (dpEnum == null) {
throw new IllegalArgumentException(
"Required TermVector Offset information was not found");
}
final String term = text.utf8ToString();
- final OffsetAttribute offsetAtt = dpEnum.attributes().getAttribute(OffsetAttribute.class);
dpEnum.nextDoc();
final int freq = dpEnum.freq();
for(int posUpto=0;posUpto<freq;posUpto++) {
final int pos = dpEnum.nextPosition();
final Token token = new Token(term,
- offsetAtt.startOffset(),
- offsetAtt.endOffset());
+ dpEnum.startOffset(),
+ dpEnum.endOffset());
if (tokenPositionsGuaranteedContiguous && pos != -1) {
// We have positions stored and a guarantee that the token position
// information is contiguous
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenStreamFromTermPositionVector.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenStreamFromTermPositionVector.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenStreamFromTermPositionVector.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/TokenStreamFromTermPositionVector.java Wed Jan 18 22:28:07 2012
@@ -60,22 +60,23 @@ public final class TokenStreamFromTermPo
BytesRef text;
DocsAndPositionsEnum dpEnum = null;
while((text = termsEnum.next()) != null) {
- dpEnum = termsEnum.docsAndPositions(null, dpEnum);
- dpEnum.nextDoc();
- final int freq = dpEnum.freq();
- final OffsetAttribute offsetAtt;
- if (dpEnum.attributes().hasAttribute(OffsetAttribute.class)) {
- offsetAtt = dpEnum.attributes().getAttribute(OffsetAttribute.class);
+ dpEnum = termsEnum.docsAndPositions(null, dpEnum, true);
+ final boolean hasOffsets;
+ if (dpEnum == null) {
+ hasOffsets = false;
+ dpEnum = termsEnum.docsAndPositions(null, dpEnum, false);
} else {
- offsetAtt = null;
+ hasOffsets = true;
}
+ dpEnum.nextDoc();
+ final int freq = dpEnum.freq();
for (int j = 0; j < freq; j++) {
int pos = dpEnum.nextPosition();
Token token;
- if (offsetAtt != null) {
+ if (hasOffsets) {
token = new Token(text.utf8ToString(),
- offsetAtt.startOffset(),
- offsetAtt.endOffset());
+ dpEnum.startOffset(),
+ dpEnum.endOffset());
} else {
token = new Token();
token.setEmpty().append(text.utf8ToString());
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/BaseFragmentsBuilder.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/BaseFragmentsBuilder.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/BaseFragmentsBuilder.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/BaseFragmentsBuilder.java Wed Jan 18 22:28:07 2012
@@ -128,8 +128,6 @@ public abstract class BaseFragmentsBuild
public void stringField(FieldInfo fieldInfo, String value) throws IOException {
FieldType ft = new FieldType(TextField.TYPE_STORED);
ft.setStoreTermVectors(fieldInfo.storeTermVector);
- ft.setStoreTermVectorOffsets(fieldInfo.storeOffsetWithTermVector);
- ft.setStoreTermVectorPositions(fieldInfo.storePositionWithTermVector);
fields.add(new Field(fieldInfo.name, value, ft));
}
@@ -168,7 +166,12 @@ public abstract class BaseFragmentsBuild
buffer.append( values[index[0]++].stringValue() );
buffer.append( getMultiValuedSeparator() );
}
- int eo = buffer.length() < endOffset ? buffer.length() : boundaryScanner.findEndOffset( buffer, endOffset );
+ int bufferLength = buffer.length();
+ // we added the multi value char to the last buffer, ignore it
+ if (values[index[0] - 1].fieldType().tokenized()) {
+ bufferLength--;
+ }
+ int eo = bufferLength < endOffset ? bufferLength : boundaryScanner.findEndOffset( buffer, endOffset );
modifiedStartOffset[0] = boundaryScanner.findStartOffset( buffer, startOffset );
return buffer.substring( modifiedStartOffset[0], eo );
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/FieldTermStack.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/FieldTermStack.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/FieldTermStack.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/FieldTermStack.java Wed Jan 18 22:28:07 2012
@@ -21,7 +21,6 @@ import java.util.Collections;
import java.util.LinkedList;
import java.util.Set;
-import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.index.DocsAndPositionsEnum;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.IndexReader;
@@ -101,29 +100,19 @@ public class FieldTermStack {
if (!termSet.contains(term)) {
continue;
}
- dpEnum = termsEnum.docsAndPositions(null, dpEnum);
+ dpEnum = termsEnum.docsAndPositions(null, dpEnum, true);
if (dpEnum == null) {
// null snippet
return;
}
- if (!dpEnum.attributes().hasAttribute(OffsetAttribute.class)) {
- // null snippet
- return;
- }
dpEnum.nextDoc();
- final OffsetAttribute offsetAtt = dpEnum.attributes().getAttribute(OffsetAttribute.class);
-
final int freq = dpEnum.freq();
for(int i = 0;i < freq;i++) {
- final int pos = dpEnum.nextPosition();
- if (pos == -1) {
- // null snippet
- return;
- }
- termList.add(new TermInfo(term, offsetAtt.startOffset(), offsetAtt.endOffset(), pos));
+ int pos = dpEnum.nextPosition();
+ termList.add(new TermInfo(term, dpEnum.startOffset(), dpEnum.endOffset(), pos));
}
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScanner.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScanner.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScanner.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScanner.java Wed Jan 18 22:28:07 2012
@@ -61,6 +61,10 @@ public class SimpleBoundaryScanner imple
if( boundaryChars.contains( buffer.charAt( offset - 1 ) ) ) return offset;
offset--;
}
+ // if we scanned up to the start of the text, return it, its a "boundary"
+ if (offset == 0) {
+ return 0;
+ }
// not found
return start;
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java Wed Jan 18 22:28:07 2012
@@ -386,7 +386,7 @@ public class HighlighterTest extends Bas
Highlighter highlighter = new Highlighter(this, scorer);
for (int i = 0; i < hits.totalHits; i++) {
- String text = searcher.doc(hits.scoreDocs[i].doc).get(NUMERIC_FIELD_NAME);
+ String text = searcher.doc(hits.scoreDocs[i].doc).getField(NUMERIC_FIELD_NAME).numericValue().toString();
TokenStream tokenStream = analyzer.tokenStream(FIELD_NAME, new StringReader(text));
highlighter.setTextFragmenter(new SimpleFragmenter(40));
@@ -1738,25 +1738,21 @@ public class HighlighterTest extends Bas
addDoc(writer, text);
}
Document doc = new Document();
- NumericField nfield = new NumericField(NUMERIC_FIELD_NAME, NumericField.TYPE_STORED);
- nfield.setIntValue(1);
- doc.add(nfield);
+ doc.add(new NumericField(NUMERIC_FIELD_NAME, 1, NumericField.getFieldType(NumericField.DataType.INT, true)));
writer.addDocument(doc, analyzer);
- nfield = new NumericField(NUMERIC_FIELD_NAME, NumericField.TYPE_STORED);
- nfield.setIntValue(3);
+
doc = new Document();
- doc.add(nfield);
+ doc.add(new NumericField(NUMERIC_FIELD_NAME, 3, NumericField.getFieldType(NumericField.DataType.INT, true)));
writer.addDocument(doc, analyzer);
- nfield = new NumericField(NUMERIC_FIELD_NAME, NumericField.TYPE_STORED);
- nfield.setIntValue(5);
+
doc = new Document();
- doc.add(nfield);
+ doc.add(new NumericField(NUMERIC_FIELD_NAME, 5, NumericField.getFieldType(NumericField.DataType.INT, true)));
writer.addDocument(doc, analyzer);
- nfield = new NumericField(NUMERIC_FIELD_NAME, NumericField.TYPE_STORED);
- nfield.setIntValue(7);
+
doc = new Document();
- doc.add(nfield);
+ doc.add(new NumericField(NUMERIC_FIELD_NAME, 7, NumericField.getFieldType(NumericField.DataType.INT, true)));
writer.addDocument(doc, analyzer);
+
writer.forceMerge(1);
writer.close();
reader = IndexReader.open(ramDir);
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/ScoreOrderFragmentsBuilderTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/ScoreOrderFragmentsBuilderTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/ScoreOrderFragmentsBuilderTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/ScoreOrderFragmentsBuilderTest.java Wed Jan 18 22:28:07 2012
@@ -35,7 +35,7 @@ public class ScoreOrderFragmentsBuilderT
String[] f = sofb.createFragments( reader, 0, F, ffl, 3 );
assertEquals( 3, f.length );
// check score order
- assertEquals( "<b>c</b> <b>a</b> <b>a</b> b b ", f[0] );
+ assertEquals( "<b>c</b> <b>a</b> <b>a</b> b b", f[0] );
assertEquals( "b b <b>a</b> b <b>a</b> b b b b b c", f[1] );
assertEquals( "<b>a</b> b b b b b b b b b b", f[2] );
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScannerTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScannerTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScannerTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScannerTest.java Wed Jan 18 22:28:07 2012
@@ -36,6 +36,10 @@ public class SimpleBoundaryScannerTest e
start = TEXT.indexOf("formance");
int expected = TEXT.indexOf("high-performance");
assertEquals(expected, scanner.findStartOffset(text, start));
+
+ start = TEXT.indexOf("che");
+ expected = TEXT.indexOf("Apache");
+ assertEquals(expected, scanner.findStartOffset(text, start));
}
public void testFindEndOffset() throws Exception {
Modified: lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleFragmentsBuilderTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleFragmentsBuilderTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleFragmentsBuilderTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/highlighter/src/test/org/apache/lucene/search/vectorhighlight/SimpleFragmentsBuilderTest.java Wed Jan 18 22:28:07 2012
@@ -37,11 +37,11 @@ public class SimpleFragmentsBuilderTest
public void test1TermIndex() throws Exception {
FieldFragList ffl = ffl(new TermQuery(new Term(F, "a")), "a" );
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
- assertEquals( "<b>a</b> ", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "<b>a</b>", sfb.createFragment( reader, 0, F, ffl ) );
// change tags
sfb = new SimpleFragmentsBuilder( new String[]{ "[" }, new String[]{ "]" } );
- assertEquals( "[a] ", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "[a]", sfb.createFragment( reader, 0, F, ffl ) );
}
public void test2Frags() throws Exception {
@@ -51,7 +51,7 @@ public class SimpleFragmentsBuilderTest
// 3 snippets requested, but should be 2
assertEquals( 2, f.length );
assertEquals( "<b>a</b> b b b b b b b b b b", f[0] );
- assertEquals( "b b <b>a</b> b <b>a</b> b ", f[1] );
+ assertEquals( "b b <b>a</b> b <b>a</b> b", f[1] );
}
public void test3Frags() throws Exception {
@@ -65,7 +65,7 @@ public class SimpleFragmentsBuilderTest
assertEquals( 3, f.length );
assertEquals( "<b>a</b> b b b b b b b b b b", f[0] );
assertEquals( "b b <b>a</b> b <b>a</b> b b b b b c", f[1] );
- assertEquals( "<b>c</b> <b>a</b> <b>a</b> b b ", f[2] );
+ assertEquals( "<b>c</b> <b>a</b> <b>a</b> b b", f[2] );
}
public void testTagsAndEncoder() throws Exception {
@@ -73,7 +73,7 @@ public class SimpleFragmentsBuilderTest
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
String[] preTags = { "[" };
String[] postTags = { "]" };
- assertEquals( "<h1> [a] </h1> ",
+ assertEquals( "<h1> [a] </h1>",
sfb.createFragment( reader, 0, F, ffl, preTags, postTags, new SimpleHTMLEncoder() ) );
}
@@ -94,7 +94,7 @@ public class SimpleFragmentsBuilderTest
SimpleFragListBuilder sflb = new SimpleFragListBuilder();
FieldFragList ffl = sflb.createFieldFragList( fpl, 100 );
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
- assertEquals( "a b c <b>d</b> e ", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "a b c <b>d</b> e", sfb.createFragment( reader, 0, F, ffl ) );
}
public void test1PhraseLongMV() throws Exception {
@@ -119,7 +119,7 @@ public class SimpleFragmentsBuilderTest
SimpleFragListBuilder sflb = new SimpleFragListBuilder();
FieldFragList ffl = sflb.createFieldFragList( fpl, 100 );
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
- assertEquals( "processing <b>speed</b>, the ", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "processing <b>speed</b>, the", sfb.createFragment( reader, 0, F, ffl ) );
}
public void testUnstoredField() throws Exception {
@@ -160,7 +160,7 @@ public class SimpleFragmentsBuilderTest
FieldFragList ffl = sflb.createFieldFragList( fpl, 100 );
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
sfb.setMultiValuedSeparator( '/' );
- assertEquals( "abc/<b>defg</b>/hijkl/", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "abc/<b>defg</b>/hijkl", sfb.createFragment( reader, 0, F, ffl ) );
}
public void testMVSeparator() throws Exception {
@@ -173,6 +173,6 @@ public class SimpleFragmentsBuilderTest
FieldFragList ffl = sflb.createFieldFragList( fpl, 100 );
SimpleFragmentsBuilder sfb = new SimpleFragmentsBuilder();
sfb.setMultiValuedSeparator( '/' );
- assertEquals( " b c//<b>d</b> e/", sfb.createFragment( reader, 0, F, ffl ) );
+ assertEquals( "//a b c//<b>d</b> e", sfb.createFragment( reader, 0, F, ffl ) );
}
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java Wed Jan 18 22:28:07 2012
@@ -21,7 +21,6 @@ import java.io.IOException;
import java.io.StringReader;
import java.util.Arrays;
import java.util.Collection;
-import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.Iterator;
@@ -34,13 +33,15 @@ import org.apache.lucene.analysis.tokena
import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;
import org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;
import org.apache.lucene.document.Document;
+import org.apache.lucene.index.Norm;
+import org.apache.lucene.index.DocValues;
import org.apache.lucene.index.DocsAndPositionsEnum;
import org.apache.lucene.index.DocsEnum;
+import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.FieldInvertState;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.FieldsEnum;
import org.apache.lucene.index.IndexReader.AtomicReaderContext;
-import org.apache.lucene.index.DocValues;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.OrdTermState;
import org.apache.lucene.index.StoredFieldVisitor;
@@ -48,7 +49,7 @@ import org.apache.lucene.index.Term;
import org.apache.lucene.index.TermState;
import org.apache.lucene.index.Terms;
import org.apache.lucene.index.TermsEnum;
-import org.apache.lucene.index.memory.MemoryIndexNormDocValues.SingleByteSource;
+import org.apache.lucene.index.memory.MemoryIndexNormDocValues.SingleValueSource;
import org.apache.lucene.search.Collector;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
@@ -197,6 +198,8 @@ public class MemoryIndex {
private static final float docBoost = 1.0f;
private static final boolean DEBUG = false;
+
+ private final FieldInfos fieldInfos;
/**
* Sorts term entries into ascending order; also works for
@@ -232,6 +235,7 @@ public class MemoryIndex {
*/
private MemoryIndex(boolean storeOffsets) {
this.stride = storeOffsets ? 3 : 1;
+ fieldInfos = new FieldInfos();
}
/**
@@ -347,6 +351,8 @@ public class MemoryIndex {
int numTokens = 0;
int numOverlapTokens = 0;
int pos = -1;
+
+ fieldInfos.addOrUpdate(fieldName, true);
TermToBytesRefAttribute termAtt = stream.getAttribute(TermToBytesRefAttribute.class);
PositionIncrementAttribute posIncrAttribute = stream.addAttribute(PositionIncrementAttribute.class);
@@ -392,7 +398,7 @@ public class MemoryIndex {
}
}
}
-
+
/**
* Creates and returns a searcher that can be used to execute arbitrary
* Lucene queries and to collect the resulting query results as hits.
@@ -766,6 +772,11 @@ public class MemoryIndex {
}
@Override
+ public FieldInfos getFieldInfos() {
+ return fieldInfos;
+ }
+
+ @Override
public int docFreq(String field, BytesRef term) {
Info info = getInfo(field);
int freq = 0;
@@ -943,7 +954,10 @@ public class MemoryIndex {
}
@Override
- public DocsAndPositionsEnum docsAndPositions(Bits liveDocs, DocsAndPositionsEnum reuse) {
+ public DocsAndPositionsEnum docsAndPositions(Bits liveDocs, DocsAndPositionsEnum reuse, boolean needsOffsets) {
+ if (needsOffsets) {
+ return null;
+ }
if (reuse == null || !(reuse instanceof MemoryDocsAndPositionsEnum)) {
reuse = new MemoryDocsAndPositionsEnum();
}
@@ -1056,6 +1070,16 @@ public class MemoryIndex {
}
@Override
+ public int startOffset() {
+ return -1;
+ }
+
+ @Override
+ public int endOffset() {
+ return -1;
+ }
+
+ @Override
public boolean hasPayload() {
return false;
}
@@ -1112,22 +1136,6 @@ public class MemoryIndex {
protected void doClose() {
if (DEBUG) System.err.println("MemoryIndexReader.doClose");
}
-
- // lucene >= 1.9 (remove this method for lucene-1.4.3)
- @Override
- public Collection<String> getFieldNames(FieldOption fieldOption) {
- if (DEBUG) System.err.println("MemoryIndexReader.getFieldNamesOption");
- if (fieldOption == FieldOption.UNINDEXED)
- return Collections.<String>emptySet();
- if (fieldOption == FieldOption.INDEXED_NO_TERMVECTOR)
- return Collections.<String>emptySet();
- if (fieldOption == FieldOption.TERMVECTOR_WITH_OFFSET && stride == 1)
- return Collections.<String>emptySet();
- if (fieldOption == FieldOption.TERMVECTOR_WITH_POSITION_OFFSET && stride == 1)
- return Collections.<String>emptySet();
-
- return Collections.unmodifiableSet(fields.keySet());
- }
@Override
public DocValues docValues(String field) throws IOException {
@@ -1150,8 +1158,9 @@ public class MemoryIndex {
int numOverlapTokens = info != null ? info.numOverlapTokens : 0;
float boost = info != null ? info.getBoost() : 1.0f;
FieldInvertState invertState = new FieldInvertState(0, numTokens, numOverlapTokens, 0, boost);
- byte norm = fieldSim.computeNorm(invertState);
- SingleByteSource singleByteSource = new SingleByteSource(new byte[] {norm});
+ Norm norm = new Norm();
+ fieldSim.computeNorm(invertState, norm);
+ SingleValueSource singleByteSource = new SingleValueSource(norm);
norms = new MemoryIndexNormDocValues(singleByteSource);
// cache it for future reuse
cachedNormValues = norms;
Modified: lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndexNormDocValues.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndexNormDocValues.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndexNormDocValues.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndexNormDocValues.java Wed Jan 18 22:28:07 2012
@@ -17,6 +17,7 @@ package org.apache.lucene.index.memory;
*/
import java.io.IOException;
+import org.apache.lucene.index.Norm;
import org.apache.lucene.index.DocValues;
import org.apache.lucene.util.BytesRef;
@@ -51,21 +52,56 @@ class MemoryIndexNormDocValues extends D
return 1;
}
- public static class SingleByteSource extends Source {
+ public static class SingleValueSource extends Source {
- private final byte[] bytes;
+ private final Number numericValue;
+ private final BytesRef binaryValue;
- protected SingleByteSource(byte[] bytes) {
- super(Type.BYTES_FIXED_STRAIGHT);
- this.bytes = bytes;
+ protected SingleValueSource(Norm norm) {
+ super(norm.type());
+ this.numericValue = norm.field().numericValue();
+ this.binaryValue = norm.field().binaryValue();
+ }
+
+ @Override
+ public long getInt(int docID) {
+ switch (type) {
+ case FIXED_INTS_16:
+ case FIXED_INTS_32:
+ case FIXED_INTS_64:
+ case FIXED_INTS_8:
+ case VAR_INTS:
+ assert numericValue != null;
+ return numericValue.longValue();
+ }
+ return super.getInt(docID);
+ }
+
+ @Override
+ public double getFloat(int docID) {
+ switch (type) {
+ case FLOAT_32:
+ case FLOAT_64:
+ assert numericValue != null;
+ return numericValue.floatValue();
+ }
+ return super.getFloat(docID);
}
@Override
public BytesRef getBytes(int docID, BytesRef ref) {
- ref.bytes = bytes;
- ref.offset = docID;
- ref.length = 1;
- return ref;
+ switch (type) {
+ case BYTES_FIXED_DEREF:
+ case BYTES_FIXED_SORTED:
+ case BYTES_FIXED_STRAIGHT:
+ case BYTES_VAR_DEREF:
+ case BYTES_VAR_SORTED:
+ case BYTES_VAR_STRAIGHT:
+ assert binaryValue != null;
+ ref.copyBytes(binaryValue);
+ return ref;
+ }
+ return super.getBytes(docID, ref);
}
@Override
@@ -75,9 +111,33 @@ class MemoryIndexNormDocValues extends D
@Override
public Object getArray() {
- return bytes;
+ switch (type) {
+ case BYTES_FIXED_DEREF:
+ case BYTES_FIXED_SORTED:
+ case BYTES_FIXED_STRAIGHT:
+ case BYTES_VAR_DEREF:
+ case BYTES_VAR_SORTED:
+ case BYTES_VAR_STRAIGHT:
+ return binaryValue.bytes;
+ case FIXED_INTS_16:
+ return new short[] { numericValue.shortValue() };
+ case FIXED_INTS_32:
+ return new int[] { numericValue.intValue() };
+ case FIXED_INTS_64:
+ return new long[] { numericValue.longValue() };
+ case FIXED_INTS_8:
+ return new byte[] { numericValue.byteValue() };
+ case VAR_INTS:
+ return new long[] { numericValue.longValue() };
+ case FLOAT_32:
+ return new float[] { numericValue.floatValue() };
+ case FLOAT_64:
+ return new double[] { numericValue.doubleValue() };
+ default:
+ throw new IllegalArgumentException("unknown type " + type);
+ }
+
}
-
}
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/memory/src/test/org/apache/lucene/index/memory/MemoryIndexTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/memory/src/test/org/apache/lucene/index/memory/MemoryIndexTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/memory/src/test/org/apache/lucene/index/memory/MemoryIndexTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/memory/src/test/org/apache/lucene/index/memory/MemoryIndexTest.java Wed Jan 18 22:28:07 2012
@@ -206,7 +206,7 @@ public class MemoryIndexTest extends Bas
MemoryIndex memory = new MemoryIndex();
memory.addField("foo", "bar", analyzer);
IndexReader reader = memory.createSearcher().getIndexReader();
- DocsAndPositionsEnum disi = reader.termPositionsEnum(null, "foo", new BytesRef("bar"));
+ DocsAndPositionsEnum disi = reader.termPositionsEnum(null, "foo", new BytesRef("bar"), false);
int docid = disi.docID();
assertTrue(docid == -1 || docid == DocIdSetIterator.NO_MORE_DOCS);
assertTrue(disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS);
@@ -214,7 +214,7 @@ public class MemoryIndexTest extends Bas
// now reuse and check again
TermsEnum te = reader.terms("foo").iterator(null);
assertTrue(te.seekExact(new BytesRef("bar"), true));
- disi = te.docsAndPositions(null, disi);
+ disi = te.docsAndPositions(null, disi, false);
docid = disi.docID();
assertTrue(docid == -1 || docid == DocIdSetIterator.NO_MORE_DOCS);
assertTrue(disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS);
Modified: lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/document/LazyDocument.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/document/LazyDocument.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/document/LazyDocument.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/document/LazyDocument.java Wed Jan 18 22:28:07 2012
@@ -23,13 +23,10 @@ import java.util.Map;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
-import org.apache.lucene.document.NumericField.DataType;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexableField;
import org.apache.lucene.index.IndexableFieldType;
-import org.apache.lucene.index.DocValue;
-import org.apache.lucene.index.DocValues;
import org.apache.lucene.util.BytesRef;
/** Defers actually loading a field's value until you ask
@@ -121,24 +118,6 @@ public class LazyDocument {
}
@Override
- public boolean numeric() {
- if (num == 0) {
- return getDocument().getField(name).numeric();
- } else {
- return getDocument().getFields(name)[num].numeric();
- }
- }
-
- @Override
- public DataType numericDataType() {
- if (num == 0) {
- return getDocument().getField(name).numericDataType();
- } else {
- return getDocument().getFields(name)[num].numericDataType();
- }
- }
-
- @Override
public Number numericValue() {
if (num == 0) {
return getDocument().getField(name).numericValue();
@@ -157,24 +136,6 @@ public class LazyDocument {
}
@Override
- public DocValue docValue() {
- if (num == 0) {
- return getDocument().getField(name).docValue();
- } else {
- return getDocument().getFields(name)[num].docValue();
- }
- }
-
- @Override
- public DocValues.Type docValueType() {
- if (num == 0) {
- return getDocument().getField(name).docValueType();
- } else {
- return getDocument().getFields(name)[num].docValueType();
- }
- }
-
- @Override
public TokenStream tokenStream(Analyzer analyzer) throws IOException {
if (num == 0) {
return getDocument().getField(name).tokenStream(analyzer);
Modified: lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java Wed Jan 18 22:28:07 2012
@@ -19,6 +19,7 @@ package org.apache.lucene.misc;
import org.apache.lucene.search.similarities.DefaultSimilarity;
import org.apache.lucene.index.FieldInvertState;
+import org.apache.lucene.index.Norm;
/**
* A similarity with a lengthNorm that provides for a "plateau" of
@@ -106,7 +107,7 @@ public class SweetSpotSimilarity extends
* discountOverlaps is true by default or true for this
* specific field. */
@Override
- public byte computeNorm(FieldInvertState state) {
+ public void computeNorm(FieldInvertState state, Norm norm) {
final int numTokens;
if (discountOverlaps)
@@ -114,7 +115,7 @@ public class SweetSpotSimilarity extends
else
numTokens = state.getLength();
- return encodeNormValue(state.getBoost() * computeLengthNorm(numTokens));
+ norm.setByte(encodeNormValue(state.getBoost() * computeLengthNorm(numTokens)));
}
/**
Modified: lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/DirectIOLinuxDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/DirectIOLinuxDirectory.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/DirectIOLinuxDirectory.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/DirectIOLinuxDirectory.java Wed Jan 18 22:28:07 2012
@@ -17,6 +17,7 @@ package org.apache.lucene.store;
* the License.
*/
+import java.io.EOFException;
import java.io.File;
import java.io.IOException;
import java.io.FileInputStream;
@@ -340,7 +341,7 @@ public class DirectIOLinuxDirectory exte
throw new IOException(ioe.getMessage() + ": " + this, ioe);
}
if (n < 0) {
- throw new IOException("eof: " + this);
+ throw new EOFException("read past EOF: " + this);
}
buffer.rewind();
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/WindowsDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/WindowsDirectory.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/WindowsDirectory.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/misc/src/java/org/apache/lucene/store/WindowsDirectory.java Wed Jan 18 22:28:07 2012
@@ -97,7 +97,7 @@ public class WindowsDirectory extends FS
}
if (bytesRead != length) {
- throw new EOFException("Read past EOF (resource: " + this + ")");
+ throw new EOFException("read past EOF: " + this);
}
}
Modified: lucene/dev/branches/solrcloud/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java Wed Jan 18 22:28:07 2012
@@ -24,12 +24,24 @@ import org.apache.lucene.search.similari
import org.apache.lucene.search.similarities.SimilarityProvider;
import org.apache.lucene.search.similarities.TFIDFSimilarity;
import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.index.Norm;
import org.apache.lucene.index.FieldInvertState;
+
/**
* Test of the SweetSpotSimilarity
*/
public class SweetSpotSimilarityTest extends LuceneTestCase {
+
+ public static float computeAndDecodeNorm(SweetSpotSimilarity decode, Similarity encode, FieldInvertState state) {
+ return decode.decodeNormValue(computeAndGetNorm(encode, state));
+ }
+
+ public static byte computeAndGetNorm(Similarity s, FieldInvertState state) {
+ Norm norm = new Norm();
+ s.computeNorm(state, norm);
+ return norm.field().numericValue().byteValue();
+ }
public void testSweetSpotComputeNorm() {
@@ -45,9 +57,13 @@ public class SweetSpotSimilarityTest ext
invertState.setBoost(1.0f);
for (int i = 1; i < 1000; i++) {
invertState.setLength(i);
+ Norm lNorm = new Norm();
+ Norm rNorm = new Norm();
+ d.computeNorm(invertState, lNorm);
+ s.computeNorm(invertState, rNorm);
assertEquals("base case: i="+i,
- d.computeNorm(invertState),
- s.computeNorm(invertState),
+ computeAndGetNorm(d, invertState),
+ computeAndGetNorm(s, invertState),
0.0f);
}
@@ -59,15 +75,15 @@ public class SweetSpotSimilarityTest ext
invertState.setLength(i);
assertEquals("3,10: spot i="+i,
1.0f,
- ss.decodeNormValue(s.computeNorm(invertState)),
+ computeAndDecodeNorm(ss, ss, invertState),
0.0f);
}
for (int i = 10; i < 1000; i++) {
invertState.setLength(i-9);
- final byte normD = d.computeNorm(invertState);
+ final byte normD = computeAndGetNorm(d, invertState);
invertState.setLength(i);
- final byte normS = s.computeNorm(invertState);
+ final byte normS = computeAndGetNorm(s, invertState);
assertEquals("3,10: 10<x : i="+i,
normD,
normS,
@@ -105,14 +121,14 @@ public class SweetSpotSimilarityTest ext
invertState.setLength(i);
assertEquals("f: 3,10: spot i="+i,
1.0f,
- ss.decodeNormValue(sp.get("foo").computeNorm(invertState)),
+ computeAndDecodeNorm(ss, sp.get("foo"), invertState),
0.0f);
}
for (int i = 10; i < 1000; i++) {
invertState.setLength(i-9);
- final byte normD = d.computeNorm(invertState);
+ final byte normD = computeAndGetNorm(d, invertState);
invertState.setLength(i);
- final byte normS = sp.get("foo").computeNorm(invertState);
+ final byte normS = computeAndGetNorm(sp.get("foo"), invertState);
assertEquals("f: 3,10: 10<x : i="+i,
normD,
normS,
@@ -122,21 +138,21 @@ public class SweetSpotSimilarityTest ext
invertState.setLength(i);
assertEquals("f: 8,13: spot i="+i,
1.0f,
- ss.decodeNormValue(sp.get("bar").computeNorm(invertState)),
+ computeAndDecodeNorm(ss, sp.get("bar"), invertState),
0.0f);
}
for (int i = 6; i <=9; i++) {
invertState.setLength(i);
assertEquals("f: 6,9: spot i="+i,
1.0f,
- ss.decodeNormValue(sp.get("yak").computeNorm(invertState)),
+ computeAndDecodeNorm(ss, sp.get("yak"), invertState),
0.0f);
}
for (int i = 13; i < 1000; i++) {
invertState.setLength(i-12);
- final byte normD = d.computeNorm(invertState);
+ final byte normD = computeAndGetNorm(d, invertState);
invertState.setLength(i);
- final byte normS = sp.get("bar").computeNorm(invertState);
+ final byte normS = computeAndGetNorm(sp.get("bar"), invertState);
assertEquals("f: 8,13: 13<x : i="+i,
normD,
normS,
@@ -144,9 +160,9 @@ public class SweetSpotSimilarityTest ext
}
for (int i = 9; i < 1000; i++) {
invertState.setLength(i-8);
- final byte normD = d.computeNorm(invertState);
+ final byte normD = computeAndGetNorm(d, invertState);
invertState.setLength(i);
- final byte normS = sp.get("yak").computeNorm(invertState);
+ final byte normS = computeAndGetNorm(sp.get("yak"), invertState);
assertEquals("f: 6,9: 9<x : i="+i,
normD,
normS,
@@ -158,8 +174,8 @@ public class SweetSpotSimilarityTest ext
for (int i = 9; i < 1000; i++) {
invertState.setLength(i);
- final byte normSS = sp.get("a").computeNorm(invertState);
- final byte normS = sp.get("b").computeNorm(invertState);
+ final byte normSS = computeAndGetNorm(sp.get("a"), invertState);
+ final byte normS = computeAndGetNorm(sp.get("b"), invertState);
assertTrue("s: i="+i+" : a="+normSS+
" < b="+normS,
normSS < normS);
Modified: lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestCartesian.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestCartesian.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestCartesian.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestCartesian.java Wed Jan 18 22:28:07 2012
@@ -23,9 +23,11 @@ import java.util.Map;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.document.Document;
+import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.NumericField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
@@ -91,6 +93,18 @@ public class TestCartesian extends Lucen
}
}
+ private static final FieldType latLongType = new FieldType();
+ static {
+ latLongType.setIndexed(true);
+ latLongType.setStored(true);
+ latLongType.setTokenized(true);
+ latLongType.setOmitNorms(true);
+ latLongType.setIndexOptions(IndexOptions.DOCS_ONLY);
+ latLongType.setNumericType(NumericField.DataType.DOUBLE);
+ latLongType.setNumericPrecisionStep(Integer.MAX_VALUE);
+ latLongType.freeze();
+ }
+
private void addPoint(IndexWriter writer, String name, double lat, double lng) throws IOException{
Document doc = new Document();
@@ -98,8 +112,8 @@ public class TestCartesian extends Lucen
doc.add(newField("name", name, TextField.TYPE_STORED));
// convert the lat / long to lucene fields
- doc.add(new NumericField(latField, Integer.MAX_VALUE, NumericField.TYPE_STORED).setDoubleValue(lat));
- doc.add(new NumericField(lngField, Integer.MAX_VALUE, NumericField.TYPE_STORED).setDoubleValue(lng));
+ doc.add(new NumericField(latField, lat, latLongType));
+ doc.add(new NumericField(lngField, lng, latLongType));
// add a default meta field to make searching all documents easy
doc.add(newField("metafile", "doc", TextField.TYPE_STORED));
@@ -107,7 +121,7 @@ public class TestCartesian extends Lucen
int ctpsize = ctps.size();
for (int i =0; i < ctpsize; i++){
CartesianTierPlotter ctp = ctps.get(i);
- doc.add(new NumericField(ctp.getTierFieldName(), Integer.MAX_VALUE, TextField.TYPE_STORED).setDoubleValue(ctp.getTierBoxId(lat,lng)));
+ doc.add(new NumericField(ctp.getTierFieldName(), ctp.getTierBoxId(lat, lng), latLongType));
doc.add(newField(geoHashPrefix, GeoHashUtils.encode(lat,lng), StringField.TYPE_STORED));
}
@@ -248,8 +262,8 @@ public class TestCartesian extends Lucen
Document d = searcher.doc(scoreDocs[i].doc);
String name = d.get("name");
- double rsLat = Double.parseDouble(d.get(latField));
- double rsLng = Double.parseDouble(d.get(lngField));
+ double rsLat = d.getField(latField).numericValue().doubleValue();
+ double rsLng = d.getField(lngField).numericValue().doubleValue();
Double geo_distance = distances.get(scoreDocs[i].doc);
double distance = DistanceUtils.getDistanceMi(lat, lng, rsLat, rsLng);
@@ -317,8 +331,8 @@ public class TestCartesian extends Lucen
for(int i =0 ; i < results; i++){
Document d = searcher.doc(scoreDocs[i].doc);
String name = d.get("name");
- double rsLat = Double.parseDouble(d.get(latField));
- double rsLng = Double.parseDouble(d.get(lngField));
+ double rsLat = d.getField(latField).numericValue().doubleValue();
+ double rsLng = d.getField(lngField).numericValue().doubleValue();
Double geo_distance = distances.get(scoreDocs[i].doc);
double distance = DistanceUtils.getDistanceMi(lat, lng, rsLat, rsLng);
@@ -389,8 +403,8 @@ public class TestCartesian extends Lucen
Document d = searcher.doc(scoreDocs[i].doc);
String name = d.get("name");
- double rsLat = Double.parseDouble(d.get(latField));
- double rsLng = Double.parseDouble(d.get(lngField));
+ double rsLat = d.getField(latField).numericValue().doubleValue();
+ double rsLng = d.getField(lngField).numericValue().doubleValue();
Double geo_distance = distances.get(scoreDocs[i].doc);
double distance = DistanceUtils.getDistanceMi(lat, lng, rsLat, rsLng);
@@ -461,8 +475,8 @@ public class TestCartesian extends Lucen
Document d = searcher.doc(scoreDocs[i].doc);
String name = d.get("name");
- double rsLat = Double.parseDouble(d.get(latField));
- double rsLng = Double.parseDouble(d.get(lngField));
+ double rsLat = d.getField(latField).numericValue().doubleValue();
+ double rsLng = d.getField(lngField).numericValue().doubleValue();
Double geo_distance = distances.get(scoreDocs[i].doc);
double distance = DistanceUtils.getDistanceMi(lat, lng, rsLat, rsLng);
Modified: lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestDistance.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestDistance.java?rev=1233096&r1=1233095&r2=1233096&view=diff
==============================================================================
--- lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestDistance.java (original)
+++ lucene/dev/branches/solrcloud/lucene/contrib/spatial/src/test/org/apache/lucene/spatial/tier/TestDistance.java Wed Jan 18 22:28:07 2012
@@ -20,17 +20,19 @@ import java.io.IOException;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.document.Document;
+import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.NumericField;
import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexReader.AtomicReaderContext;
+import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
-import org.apache.lucene.index.IndexReader;
-import org.apache.lucene.search.QueryWrapperFilter;
import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.QueryWrapperFilter;
+import org.apache.lucene.store.Directory;
import org.apache.lucene.util.LuceneTestCase;
import org.apache.lucene.util.ReaderUtil;
-import org.apache.lucene.store.Directory;
public class TestDistance extends LuceneTestCase {
@@ -58,6 +60,18 @@ public class TestDistance extends Lucene
directory.close();
super.tearDown();
}
+
+ private static final FieldType latLongType = new FieldType();
+ static {
+ latLongType.setIndexed(true);
+ latLongType.setStored(true);
+ latLongType.setTokenized(true);
+ latLongType.setOmitNorms(true);
+ latLongType.setIndexOptions(IndexOptions.DOCS_ONLY);
+ latLongType.setNumericType(NumericField.DataType.DOUBLE);
+ latLongType.setNumericPrecisionStep(Integer.MAX_VALUE);
+ latLongType.freeze();
+ }
private void addPoint(IndexWriter writer, String name, double lat, double lng) throws IOException{
@@ -66,8 +80,8 @@ public class TestDistance extends Lucene
doc.add(newField("name", name, TextField.TYPE_STORED));
// convert the lat / long to lucene fields
- doc.add(new NumericField(latField, Integer.MAX_VALUE, NumericField.TYPE_STORED).setDoubleValue(lat));
- doc.add(new NumericField(lngField, Integer.MAX_VALUE, NumericField.TYPE_STORED).setDoubleValue(lng));
+ doc.add(new NumericField(latField, lat, latLongType));
+ doc.add(new NumericField(lngField, lng, latLongType));
// add a default meta field to make searching all documents easy
doc.add(newField("metafile", "doc", TextField.TYPE_STORED));