stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[cyang2010 <ys...@hotmail.com>]

I am using solr 1.4.1.   I am trying to index a spanish field using the
following tokenizer/filters:

    
      
        
        
        
        
	

Using field analysis solr Admin i can tell StopFilterFactory and
SnowballPorterFilterFactory with Spanish not working right:

1. after stopFilter, "la" should be gone, but it is not.
2. after snowballporterFilterFactory(language=Spanish), "cöcktäils" should
become "cöcktäil".  But i still see the token "cöcktäils" coming out.

I configured a spanish stopword list for the StopFilterFactory.

Field name: title_name
field value:  la Cöcktäils


Index Analyzer
=========================================================================
org.apache.solr.analysis.WhitespaceTokenizerFactory {}
term position 	1	2
term text 	la	Cöcktäils
term type 	word	word
source start,end 	0,2	3,12
payload 		

=============================================================================
org.apache.solr.analysis.StopFilterFactory {words=stopwords_es.txt,
ignoreCase=true}
term position 	1	2
term text 	la	Cöcktäils
term type 	word	word
source start,end 	0,2	3,12
payload 
==============================================================================	
org.apache.solr.analysis.LowerCaseFilterFactory {}
term position 	1	2
term text 	la	cöcktäils
term type 	word	word
source start,end 	0,2	3,12
payload
=============================================================================== 		
org.apache.solr.analysis.SnowballPorterFilterFactory {language=Spanish}
term position 	1	2
term text 	la	cöcktäils
term type 	word	word
source start,end 	0,2	3,12
payload 		
===============================================================================
org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory {}
term position 	1	2
term text 	la	cöcktäils
term type 	word	word
source start,end 	0,2	3,12
payload 		
==============================================================================



I just copied the text from this URL to form my stopwords_es.txt:

http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/resources/org/apache/lucene/analysis/snowball/spanish_stop.txt



Look forward to your help...

--
View this message in context: http://lucene.472066.n3.nabble.com/stopFilterFactor-and-SnowballPorterFilterFactory-not-work-for-Spanish-tp2684322p2684322.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[cyang2010 <ys...@hotmail.com>]

I just tried with some real spanish text:

"Alquileres"  -->

=============================================================
org.apache.solr.analysis.LowerCaseFilterFactory {}
term position 	1
term text 	alquileres
term type 	word
source start,end 	4,14
payload 	
============================================================
org.apache.solr.analysis.SnowballPorterFilterFactory {language=Spanish}
term position 	1
term text 	alquiler
term type 	word
source start,end 	4,14
payload 	



Looks like that spanish stemmer is working.   Thanks,

cyang

--
View this message in context: http://lucene.472066.n3.nabble.com/stopFilterFactor-and-SnowballPorterFilterFactory-not-work-for-Spanish-tp2684322p2684814.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[cyang2010 <ys...@hotmail.com>]

Sorry Robert.  I just use some text translated by someone.  Maybe that
translation is not right.

Could you please give me a spanish term which i can show the spanish
stemming factory is working?

Thanks,


cyang

--
View this message in context: http://lucene.472066.n3.nabble.com/stopFilterFactor-and-SnowballPorterFilterFactory-not-work-for-Spanish-tp2684322p2684775.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[Robert Muir <rc...@gmail.com>]

On Tue, Mar 15, 2011 at 8:50 PM, cyang2010 <ys...@hotmail.com> wrote:
> Robert,
>
> Thanks for your advice.  I modified my stopword text file.  Now the
> stopwordFilter start to work.
>
> But the stemming related filter (SnowballPorterFilterFactory-- spanish)
> still not working.  Anyone have any idea on that?
>

how is it not working? how is "cöcktäils" a spanish word!

Re: stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[cyang2010 <ys...@hotmail.com>]

Robert,

Thanks for your advice.  I modified my stopword text file.  Now the
stopwordFilter start to work.

But the stemming related filter (SnowballPorterFilterFactory-- spanish)
still not working.  Anyone have any idea on that?

Thanks,


cyang

--
View this message in context: http://lucene.472066.n3.nabble.com/stopFilterFactor-and-SnowballPorterFilterFactory-not-work-for-Spanish-tp2684322p2684713.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: stopFilterFactor and SnowballPorterFilterFactory not work for Spanish

[Robert Muir <rc...@gmail.com>]

On Tue, Mar 15, 2011 at 7:07 PM, cyang2010 <ys...@hotmail.com> wrote:

> I just copied the text from this URL to form my stopwords_es.txt:
>
> http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/resources/org/apache/lucene/analysis/snowball/spanish_stop.txt
>

you cannot use the files from analysis/snowball/* directly with Solr.
These files are not in a format that Solr understands... so you will
have to modify them (e.g. the comment character for solr is # not |,
and there can only be one word per line)