You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Darren Govoni <da...@ontrenet.com> on 2010/06/30 21:47:53 UTC
Bizarre Terms revisited
Hi,
I really think there is something "not quite right" going on here
after much study. Here is my findings.
Using MLT, I get terms that appear to be long concatenations of words
that are space delimited in the original text.
I can't think of any reason for these sentence-like terms to exist (see
below).
All my data and config follows:
Here is the output from MLT:
<lst name="interestingTerms">
<float name="text_t:result">1.0</float>
<float name="text_t:concepts">1.0</float>
<float name="text_t:identified">1.0</float>
<float name="text_t:row">1.0</float>
<float name="text_t:based">1.0</float>
<float name="text_t:000">1.0</float>
<float name="text_t:ontreweb">1.0</float>
<float name="text_t:in">1.0</float>
<float name="text_t:and">1.0</float>
<float name="text_t:2">1.0</float>
<!-- These do not look like valid or useful terms to have in the index.
-->
<!-- Why do these exist? -->
<float
name="text_t:searchinonelanguagefindresultsinanother">1.0</float>
<float name="text_t:ontrewebstartpage">1.0</float>
<float name="text_t:unlimitedmutliwordandphrasematching">1.0</float>
<float name="text_t:wordsandphrases">1.0</float>
<float name="text_t:pluggablevocabulariesontologies">1.0</float>
<float name="text_t:mappedconcepts">1.0</float>
<float name="text_t:ontrewebproductfeatures">1.0</float>
<float name="text_t:multilinguallexiconsfrenchenglishetc">1.0</float>
<float name="text_t:multipleinheritanceofconcepts">1.0</float>
<float name="text_t:4">1.0</float>
<float name="text_t:string">1.0</float>
<float name="text_t:english">1.0</float>
<float name="text_t:mapped">1.0</float>
<float name="text_t:multilingual">1.0</float>
<float name="text_t:mutliword">1.0</float>
</lst>
My field:
<field name="text_t" type="textgen" indexed="true" stored="true"
multiValued="true" termVectors="true" termPositions="true"
termOffsets="true"/>
Field definition taken from the default schema.xml
<fieldType name="textgen" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory"
synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Original text (partially snipped) as it appears in the stored index.
"Ontreweb Product Features
Unlimited mutliword and phrase matching Multiple inheritance of concepts Pluggable vocabularies, ontologies Multilingual
lexicons: french, english, etc. Search in one language, find results in another 200,000+ words and phrases, 35,000 mapped
concepts.
1. 2. 3. 4."
Re: Bizarre Terms revisited
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Bizarre Terms revisited
to clarify for folks: this is a followup to this previous thread...
http://search.lucidimagination.com/search/document/c0aabe47aad1ca3c/bizarre_tfv_output#de3abb42754407d6
: Using MLT, I get terms that appear to be long concatenations of words
: that are space delimited in the original text.
: I can't think of any reason for these sentence-like terms to exist (see
: below).
You never answered the questions i asked in the previous thread...
http://search.lucidimagination.com/search/document/c0aabe47aad1ca3c/bizarre_tfv_output#ed49cebdd92db674
>> Did you try pasting that text into the analysis page to see exactly what
>> your "text_t" field does with it at analysis time like ia suggested?
>>
>> My best hunch is that the "spaces" are not your typical basic "space"
>> character (hex 20) and maybe the tokenizer you are using doesn't
>> tokenize on them, but then perhaps something like word delimiter treats
>> them as non-word characters and chews them up.
...
>> (Tip: if you use the JSON response writer (wt=json) when looking at the
>> stored field value, it will help you see exactly what characters were in
>> the original values by showing you the unicode escapes)
FWIW: I cut/pasted the text you provided...
: Original text (partially snipped) as it appears in the stored index.
:
: "Ontreweb Product Features
:
:
:
: Unlimited mutliword and phrase matching Multiple inheritance of concepts Pluggable vocabularies, ontologies Multilingual
: lexicons: french, english, etc. Search in one language, find results in another 200,000+ words and phrases, 35,000 mapped
: concepts.
:
: 1. 2. 3. 4."
...into the example/exampledocs/solr.xml file in Solr 1.4.1 using the
field name "attr_darren" (which uses the "textgen" field type with an
analysis chain matching what you included in your last mail). When I
indexed that doc (and nothing else), and look at the list of terms indexed
in that field using the LukeRequestHandler, i get the oupput below.
In short, i can't reproduce what you are describing at all .. and my best
guess at what you are seeing (baring the possibility that this is old data
from when the field type was something else) is that what you think is
whitespace isn't actaully a space character that the WhitespaceTokenizer
recognizes.
look at the JSON output from your actual stored value, and verify that
it's not some funky UTF8 character.
http://localhost:8983/solr/admin/luke?fl=attr_darren&numTerms=1000
...
<lst name="topTerms">
<int name="1">1</int>
<int name="2">1</int>
<int name="200">1</int>
<int name="200000">1</int>
<int name="3">1</int>
<int name="35">1</int>
<int name="35000">1</int>
<int name="4">1</int>
<int name="another">1</int>
<int name="concepts">1</int>
<int name="english">1</int>
<int name="etc">1</int>
<int name="features">1</int>
<int name="find">1</int>
<int name="french">1</int>
<int name="inheritance">1</int>
<int name="language">1</int>
<int name="lexicons">1</int>
<int name="mapped">1</int>
<int name="matching">1</int>
<int name="multilingual">1</int>
<int name="multiple">1</int>
<int name="mutliword">1</int>
<int name="one">1</int>
<int name="ontologies">1</int>
<int name="ontreweb">1</int>
<int name="phrase">1</int>
<int name="phrases">1</int>
<int name="pluggable">1</int>
<int name="product">1</int>
<int name="results">1</int>
<int name="search">1</int>
<int name="unlimited">1</int>
<int name="vocabularies">1</int>
<int name="words">1</int>
<int name="000">1</int>
</lst>
-Hoss