You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Darren Govoni <da...@ontrenet.com> on 2010/06/30 21:47:53 UTC

Bizarre Terms revisited

Hi,
  I really think there is something "not quite right" going on here
after much study. Here is my findings.

Using MLT, I get terms that appear to be long concatenations of words
that are space delimited in the original text.
I can't think of any reason for these sentence-like terms to exist  (see
below).

All my data and config follows:

Here is the output from MLT:

<lst name="interestingTerms"> 
<float name="text_t:result">1.0</float> 
<float name="text_t:concepts">1.0</float> 
<float name="text_t:identified">1.0</float> 
<float name="text_t:row">1.0</float> 
<float name="text_t:based">1.0</float> 
<float name="text_t:000">1.0</float>  
<float name="text_t:ontreweb">1.0</float> 
<float name="text_t:in">1.0</float> 
<float name="text_t:and">1.0</float> 
<float name="text_t:2">1.0</float> 

<!-- These do not look like valid or useful terms to have in the index.
-->
<!-- Why do these exist? -->
<float
name="text_t:searchinonelanguagefindresultsinanother">1.0</float> 
<float name="text_t:ontrewebstartpage">1.0</float> 
<float name="text_t:unlimitedmutliwordandphrasematching">1.0</float> 
<float name="text_t:wordsandphrases">1.0</float> 
<float name="text_t:pluggablevocabulariesontologies">1.0</float> 
<float name="text_t:mappedconcepts">1.0</float> 
<float name="text_t:ontrewebproductfeatures">1.0</float> 
<float name="text_t:multilinguallexiconsfrenchenglishetc">1.0</float> 
<float name="text_t:multipleinheritanceofconcepts">1.0</float> 

<float name="text_t:4">1.0</float> 
<float name="text_t:string">1.0</float> 
<float name="text_t:english">1.0</float> 
<float name="text_t:mapped">1.0</float> 
<float name="text_t:multilingual">1.0</float> 
<float name="text_t:mutliword">1.0</float> 
</lst> 

My field:

   <field name="text_t"  type="textgen"    indexed="true"  stored="true"
multiValued="true" termVectors="true" termPositions="true"
termOffsets="true"/> 


Field definition taken from the default schema.xml

    <fieldType name="textgen" class="solr.TextField"
positionIncrementGap="100"> 
      <analyzer type="index"> 
        <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
        <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" /> 
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="0"/> 
        <filter class="solr.LowerCaseFilterFactory"/> 
      </analyzer> 
      <analyzer type="query"> 
        <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
        <filter class="solr.SynonymFilterFactory"
synonyms="synonyms.txt" ignoreCase="true" expand="true"/> 
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                /> 
        <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"/> 
        <filter class="solr.LowerCaseFilterFactory"/> 
      </analyzer> 
    </fieldType> 

Original text (partially snipped) as it appears in the stored index.

"Ontreweb Product Features 

     

Unlimited mutliword and phrase matching Multiple inheritance of concepts Pluggable vocabularies, ontologies Multilingual 
lexicons: french, english, etc. Search in one language, find results in another 200,000+ words and phrases, 35,000 mapped 
concepts.

1. 2. 3. 4."



Re: Bizarre Terms revisited

Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Bizarre Terms revisited

to clarify for folks: this is a followup to this previous thread...

http://search.lucidimagination.com/search/document/c0aabe47aad1ca3c/bizarre_tfv_output#de3abb42754407d6

: Using MLT, I get terms that appear to be long concatenations of words
: that are space delimited in the original text.
: I can't think of any reason for these sentence-like terms to exist  (see
: below).

You never answered the questions i asked in the previous thread...

http://search.lucidimagination.com/search/document/c0aabe47aad1ca3c/bizarre_tfv_output#ed49cebdd92db674
>> Did you try pasting that text into the analysis page to see exactly what 
>> your "text_t" field does with it at analysis time like ia suggested?
>> 
>> My best hunch is that the "spaces" are not your typical basic "space" 
>> character (hex 20) and maybe the tokenizer you are using doesn't 
>> tokenize on them, but then perhaps something like word delimiter treats 
>> them as non-word characters and chews them up.
	...
>> (Tip: if you use the JSON response writer (wt=json) when looking at the 
>> stored field value, it will help you see exactly what characters were in 
>> the original values by showing you the unicode escapes)

FWIW: I cut/pasted the text you provided...

: Original text (partially snipped) as it appears in the stored index.
: 
: "Ontreweb Product Features 
: 
:      
: 
: Unlimited mutliword and phrase matching Multiple inheritance of concepts Pluggable vocabularies, ontologies Multilingual 
: lexicons: french, english, etc. Search in one language, find results in another 200,000+ words and phrases, 35,000 mapped 
: concepts.
: 
: 1. 2. 3. 4."

...into the example/exampledocs/solr.xml file in Solr 1.4.1 using the 
field name "attr_darren" (which uses the "textgen" field type with an 
analysis chain matching what you included in your last mail).  When I 
indexed that doc (and nothing else), and look at the list of terms indexed 
in that field using the LukeRequestHandler, i get the oupput below.

In short, i can't reproduce what you are describing at all .. and my best 
guess at what you are seeing (baring the possibility that this is old data 
from when the field type was something else) is that what you think is 
whitespace isn't actaully a space character that the WhitespaceTokenizer 
recognizes.

look at the JSON output from your actual stored value, and verify that 
it's not some funky UTF8 character.


http://localhost:8983/solr/admin/luke?fl=attr_darren&numTerms=1000
  
  ...
  <lst name="topTerms">
	<int name="1">1</int>
	<int name="2">1</int>
	<int name="200">1</int>

	<int name="200000">1</int>
	<int name="3">1</int>
	<int name="35">1</int>
	<int name="35000">1</int>
	<int name="4">1</int>
	<int name="another">1</int>

	<int name="concepts">1</int>
	<int name="english">1</int>
	<int name="etc">1</int>
	<int name="features">1</int>
	<int name="find">1</int>
	<int name="french">1</int>

	<int name="inheritance">1</int>
	<int name="language">1</int>
	<int name="lexicons">1</int>
	<int name="mapped">1</int>
	<int name="matching">1</int>
	<int name="multilingual">1</int>

	<int name="multiple">1</int>
	<int name="mutliword">1</int>
	<int name="one">1</int>
	<int name="ontologies">1</int>
	<int name="ontreweb">1</int>
	<int name="phrase">1</int>

	<int name="phrases">1</int>
	<int name="pluggable">1</int>
	<int name="product">1</int>
	<int name="results">1</int>
	<int name="search">1</int>
	<int name="unlimited">1</int>

	<int name="vocabularies">1</int>
	<int name="words">1</int>
	<int name="000">1</int>
  </lst>









-Hoss