You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2014/02/26 19:05:52 UTC

Analyzer Does Not Works As Accepted

Hi;

I have impelented that custom Analyzer:

public class DisambiguatorAnalyzer extends Analyzer {

   Version version = Version.LUCENE_46;
   List<String> stopWordList;

   public DisambiguatorAnalyzer(List<String> stopWordList) throws
IOException {
      super();
      this.stopWordList = stopWordList;
   }

   @Override
   protected TokenStreamComponents createComponents(String fieldName,
Reader reader) {
      Tokenizer source = new WhitespaceTokenizer(version, reader);
      int flags = GENERATE_WORD_PARTS | CATENATE_WORDS;
      TokenStream filter = new WordDelimiterFilter(source,
WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE, flags, null);
      filter = new StopFilter(version, filter,
StopFilter.makeStopSet(version, stopWordList));
      filter = new TurkishLowerCaseFilter(filter);
      return new TokenStreamComponents(source, filter);
   }
}

However it preserve originals and does not remove Stopwords. What maybe the
wrong?

Thanks;
Furkan KAMACI

Re: Analyzer Does Not Works As Accepted

Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;

I have put StopFilter after lower case filter.

Thanks;
Furkan KAMACI
28 Şub 2014 12:06 tarihinde "pravesh" <su...@yahoo.com> yazdı:

> >>I fixed the problems. StopFilter was not working as accepted because of
> letter cases.
>
> Alternatively, you could have moved StopFilter above the
> WordDelimiterFilter
> in your analysis chain.
>
>
> Regards
> Pravesh
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Analyzer-Does-Not-Works-As-Accepted-tp4119898p4120363.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Analyzer Does Not Works As Accepted

Posted by pravesh <su...@yahoo.com>.
>>I fixed the problems. StopFilter was not working as accepted because of
letter cases.

Alternatively, you could have moved StopFilter above the WordDelimiterFilter
in your analysis chain.


Regards
Pravesh



--
View this message in context: http://lucene.472066.n3.nabble.com/Analyzer-Does-Not-Works-As-Accepted-tp4119898p4120363.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Analyzer Does Not Works As Accepted

Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;

I fixed the problems. StopFilter was not working as accepted because of
letter cases. I've changed the flags of WordDelimiter. Also I've
changed TokenStream
to TokenFilter.

Thanks;
Furkan KAMACI


2014-02-26 20:05 GMT+02:00 Furkan KAMACI <fu...@gmail.com>:

> Hi;
>
> I have impelented that custom Analyzer:
>
> public class DisambiguatorAnalyzer extends Analyzer {
>
>    Version version = Version.LUCENE_46;
>    List<String> stopWordList;
>
>    public DisambiguatorAnalyzer(List<String> stopWordList) throws
> IOException {
>       super();
>       this.stopWordList = stopWordList;
>    }
>
>    @Override
>    protected TokenStreamComponents createComponents(String fieldName,
> Reader reader) {
>       Tokenizer source = new WhitespaceTokenizer(version, reader);
>       int flags = GENERATE_WORD_PARTS | CATENATE_WORDS;
>       TokenStream filter = new WordDelimiterFilter(source,
> WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE, flags, null);
>       filter = new StopFilter(version, filter,
> StopFilter.makeStopSet(version, stopWordList));
>       filter = new TurkishLowerCaseFilter(filter);
>       return new TokenStreamComponents(source, filter);
>    }
> }
>
> However it preserve originals and does not remove Stopwords. What maybe
> the wrong?
>
> Thanks;
> Furkan KAMACI
>