You are viewing a plain text version of this content. The canonical link for it is here.

Posted to java-user@lucene.apache.org by Kumaran Ramasubramanian <ku...@gmail.com> on 2016/10/11 14:21:49 UTC

How to add ASCIIFoldingFilter in ClassicAnalyzer

Hi All,

  Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without
writing a new custom analyzer ? should i extend StopwordAnalyzerBase again?


I know that ClassicAnalyzer is final. any special purpose for making it as
final? Because, StandardAnalyzer was not final before ?

public final class ClassicAnalyzer extends StopwordAnalyzerBase
>


--
Kumaran R

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Kumaran Ramasubramanian <ku...@gmail.com>.

Hi Adrien

Thanks a lot for the pointer.


--
Kumaran R


On Wed, Oct 19, 2016 at 8:07 PM, Adrien Grand <jp...@gmail.com> wrote:

> You would need to override the wrapComponents method in order to wrap the
> tokenstream. See for instance Lucene's LimitTokenCountAnalyzer.
>
> Le mar. 18 oct. 2016 à 18:46, Kumaran Ramasubramanian <ku...@gmail.com>
> a écrit :
>
> > Hi Adrien
> >
> > How to do this? Any Pointers?
> >
> > 
> > > If it is fine to add the ascii folding filter at the end of the
> analysis
> >
> > chain, then you could use AnalyzerWrapper. 
> > >
> >
> >
> >
> >
> > -
> > Kumaran R
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian <
> > kums.134@gmail.com
> > > wrote:
> >
> > >
> > >
> > > @Ahmet, Uwe: Thanks a lot for your suggestion. Already i have written
> > > custom analyzer as you said. But just trying to avoid new component in
> my
> > > search flow.
> > >
> > > @Adrien: how to add filter using AnalyzerWrapper. Any pointers?
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Oct 11, 2016 at 8:16 PM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> > >
> > >> I'd suggest to use CustomAnalyzer for defining your own analyzer. This
> > >> allows to build your own analyzer with the components (tokenizers and
> > >> filters) you like to have.
> > >>
> > >> Uwe
> > >>
> > >> -----
> > >> Uwe Schindler
> > >> H.-H.-Meier-Allee 63, D-28213 Bremen
> > >> http://www.thetaphi.de
> > >> eMail: uwe@thetaphi.de
> > >>
> > >> > -----Original Message-----
> > >> > From: Adrien Grand [mailto:jpountz@gmail.com]
> > >> > Sent: Tuesday, October 11, 2016 4:37 PM
> > >> > To: java-user@lucene.apache.org
> > >> > Subject: Re: How to add ASCIIFoldingFilter in ClassicAnalyzer
> > >> >
> > >> > Hi Kumaran,
> > >> >
> > >> > If it is fine to add the ascii folding filter at the end of the
> > analysis
> > >> > chain, then you could use AnalyzerWrapper. Otherwise, you need to
> > >> create a
> > >> > new analyzer that has the same analysis chain as ClassicAnalyzer,
> plus
> > >> an
> > >> > ASCIIFoldingFilter.
> > >> >
> > >> > Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian
> > >> > <ku...@gmail.com>
> > >> > a écrit :
> > >> >
> > >> > > Hi All,
> > >> > >
> > >> > >   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer
> > >> without
> > >> > > writing a new custom analyzer ? should i extend
> StopwordAnalyzerBase
> > >> > again?
> > >> > >
> > >> > >
> > >> > > I know that ClassicAnalyzer is final. any special purpose for
> making
> > >> it as
> > >> > > final? Because, StandardAnalyzer was not final before ?
> > >> > >
> > >> > > public final class ClassicAnalyzer extends StopwordAnalyzerBase
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Kumaran R
> > >> > >
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>
> > >>
> > >
> >
>

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Adrien Grand <jp...@gmail.com>.

You would need to override the wrapComponents method in order to wrap the
tokenstream. See for instance Lucene's LimitTokenCountAnalyzer.

Le mar. 18 oct. 2016 à 18:46, Kumaran Ramasubramanian <ku...@gmail.com>
a écrit :

> Hi Adrien
>
> How to do this? Any Pointers?
>
> 
> > If it is fine to add the ascii folding filter at the end of the analysis
>
> chain, then you could use AnalyzerWrapper. 
> >
>
>
>
>
> -
> Kumaran R
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian <
> kums.134@gmail.com
> > wrote:
>
> >
> >
> > @Ahmet, Uwe: Thanks a lot for your suggestion. Already i have written
> > custom analyzer as you said. But just trying to avoid new component in my
> > search flow.
> >
> > @Adrien: how to add filter using AnalyzerWrapper. Any pointers?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 8:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> >> I'd suggest to use CustomAnalyzer for defining your own analyzer. This
> >> allows to build your own analyzer with the components (tokenizers and
> >> filters) you like to have.
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: uwe@thetaphi.de
> >>
> >> > -----Original Message-----
> >> > From: Adrien Grand [mailto:jpountz@gmail.com]
> >> > Sent: Tuesday, October 11, 2016 4:37 PM
> >> > To: java-user@lucene.apache.org
> >> > Subject: Re: How to add ASCIIFoldingFilter in ClassicAnalyzer
> >> >
> >> > Hi Kumaran,
> >> >
> >> > If it is fine to add the ascii folding filter at the end of the
> analysis
> >> > chain, then you could use AnalyzerWrapper. Otherwise, you need to
> >> create a
> >> > new analyzer that has the same analysis chain as ClassicAnalyzer, plus
> >> an
> >> > ASCIIFoldingFilter.
> >> >
> >> > Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian
> >> > <ku...@gmail.com>
> >> > a écrit :
> >> >
> >> > > Hi All,
> >> > >
> >> > >   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer
> >> without
> >> > > writing a new custom analyzer ? should i extend StopwordAnalyzerBase
> >> > again?
> >> > >
> >> > >
> >> > > I know that ClassicAnalyzer is final. any special purpose for making
> >> it as
> >> > > final? Because, StandardAnalyzer was not final before ?
> >> > >
> >> > > public final class ClassicAnalyzer extends StopwordAnalyzerBase
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Kumaran R
> >> > >
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Kumaran Ramasubramanian <ku...@gmail.com>.

Hi Adrien

How to do this? Any Pointers?


> If it is fine to add the ascii folding filter at the end of the analysis

chain, then you could use AnalyzerWrapper. 
>




-
Kumaran R









On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian <kums.134@gmail.com
> wrote:

>
>
> @Ahmet, Uwe: Thanks a lot for your suggestion. Already i have written
> custom analyzer as you said. But just trying to avoid new component in my
> search flow.
>
> @Adrien: how to add filter using AnalyzerWrapper. Any pointers?
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 8:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> I'd suggest to use CustomAnalyzer for defining your own analyzer. This
>> allows to build your own analyzer with the components (tokenizers and
>> filters) you like to have.
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>> > -----Original Message-----
>> > From: Adrien Grand [mailto:jpountz@gmail.com]
>> > Sent: Tuesday, October 11, 2016 4:37 PM
>> > To: java-user@lucene.apache.org
>> > Subject: Re: How to add ASCIIFoldingFilter in ClassicAnalyzer
>> >
>> > Hi Kumaran,
>> >
>> > If it is fine to add the ascii folding filter at the end of the analysis
>> > chain, then you could use AnalyzerWrapper. Otherwise, you need to
>> create a
>> > new analyzer that has the same analysis chain as ClassicAnalyzer, plus
>> an
>> > ASCIIFoldingFilter.
>> >
>> > Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian
>> > <ku...@gmail.com>
>> > a écrit :
>> >
>> > > Hi All,
>> > >
>> > >   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer
>> without
>> > > writing a new custom analyzer ? should i extend StopwordAnalyzerBase
>> > again?
>> > >
>> > >
>> > > I know that ClassicAnalyzer is final. any special purpose for making
>> it as
>> > > final? Because, StandardAnalyzer was not final before ?
>> > >
>> > > public final class ClassicAnalyzer extends StopwordAnalyzerBase
>> > > >
>> > >
>> > >
>> > > --
>> > > Kumaran R
>> > >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Kumaran Ramasubramanian <ku...@gmail.com>.

@Ahmet, Uwe: Thanks a lot for your suggestion. Already i have written
custom analyzer as you said. But just trying to avoid new component in my
search flow.

@Adrien: how to add filter using AnalyzerWrapper. Any pointers?









On Tue, Oct 11, 2016 at 8:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:

> I'd suggest to use CustomAnalyzer for defining your own analyzer. This
> allows to build your own analyzer with the components (tokenizers and
> filters) you like to have.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Adrien Grand [mailto:jpountz@gmail.com]
> > Sent: Tuesday, October 11, 2016 4:37 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: How to add ASCIIFoldingFilter in ClassicAnalyzer
> >
> > Hi Kumaran,
> >
> > If it is fine to add the ascii folding filter at the end of the analysis
> > chain, then you could use AnalyzerWrapper. Otherwise, you need to create
> a
> > new analyzer that has the same analysis chain as ClassicAnalyzer, plus an
> > ASCIIFoldingFilter.
> >
> > Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian
> > <ku...@gmail.com>
> > a écrit :
> >
> > > Hi All,
> > >
> > >   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer
> without
> > > writing a new custom analyzer ? should i extend StopwordAnalyzerBase
> > again?
> > >
> > >
> > > I know that ClassicAnalyzer is final. any special purpose for making
> it as
> > > final? Because, StandardAnalyzer was not final before ?
> > >
> > > public final class ClassicAnalyzer extends StopwordAnalyzerBase
> > > >
> > >
> > >
> > > --
> > > Kumaran R
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Uwe Schindler <uw...@thetaphi.de>.

I'd suggest to use CustomAnalyzer for defining your own analyzer. This allows to build your own analyzer with the components (tokenizers and filters) you like to have.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Tuesday, October 11, 2016 4:37 PM
> To: java-user@lucene.apache.org
> Subject: Re: How to add ASCIIFoldingFilter in ClassicAnalyzer
> 
> Hi Kumaran,
> 
> If it is fine to add the ascii folding filter at the end of the analysis
> chain, then you could use AnalyzerWrapper. Otherwise, you need to create a
> new analyzer that has the same analysis chain as ClassicAnalyzer, plus an
> ASCIIFoldingFilter.
> 
> Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian
> <ku...@gmail.com>
> a écrit :
> 
> > Hi All,
> >
> >   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without
> > writing a new custom analyzer ? should i extend StopwordAnalyzerBase
> again?
> >
> >
> > I know that ClassicAnalyzer is final. any special purpose for making it as
> > final? Because, StandardAnalyzer was not final before ?
> >
> > public final class ClassicAnalyzer extends StopwordAnalyzerBase
> > >
> >
> >
> > --
> > Kumaran R
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Adrien Grand <jp...@gmail.com>.

Hi Kumaran,

If it is fine to add the ascii folding filter at the end of the analysis
chain, then you could use AnalyzerWrapper. Otherwise, you need to create a
new analyzer that has the same analysis chain as ClassicAnalyzer, plus an
ASCIIFoldingFilter.

Le mar. 11 oct. 2016 à 16:22, Kumaran Ramasubramanian <ku...@gmail.com>
a écrit :

> Hi All,
>
>   Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without
> writing a new custom analyzer ? should i extend StopwordAnalyzerBase again?
>
>
> I know that ClassicAnalyzer is final. any special purpose for making it as
> final? Because, StandardAnalyzer was not final before ?
>
> public final class ClassicAnalyzer extends StopwordAnalyzerBase
> >
>
>
> --
> Kumaran R
>

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.

Hi,

I forgot to include : .addTokenFilter("asciifolding")

Ahmet


On Tuesday, October 11, 2016 5:37 PM, Ahmet Arslan <io...@yahoo.com> wrote:
Hi Kumaran,

Writing a custom analyzer is easier than it seems.

Please see how I added kstem to classic analyzer:

return CustomAnalyzer.builder()
.withTokenizer("classic")
.addTokenFilter("classic")
.addTokenFilter("lowercase")
.addTokenFilter("kstem")
.build();

Ahmet




On Tuesday, October 11, 2016 5:22 PM, Kumaran Ramasubramanian <ku...@gmail.com> wrote:
Hi All,

  Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without
writing a new custom analyzer ? should i extend StopwordAnalyzerBase again?


I know that ClassicAnalyzer is final. any special purpose for making it as
final? Because, StandardAnalyzer was not final before ?

public final class ClassicAnalyzer extends StopwordAnalyzerBase
>


--
Kumaran R

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.

Hi Kumaran,

Writing a custom analyzer is easier than it seems.

Please see how I added kstem to classic analyzer:

return CustomAnalyzer.builder()
.withTokenizer("classic")
.addTokenFilter("classic")
.addTokenFilter("lowercase")
.addTokenFilter("kstem")
.build();

Ahmet



On Tuesday, October 11, 2016 5:22 PM, Kumaran Ramasubramanian <ku...@gmail.com> wrote:
Hi All,

  Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without
writing a new custom analyzer ? should i extend StopwordAnalyzerBase again?


I know that ClassicAnalyzer is final. any special purpose for making it as
final? Because, StandardAnalyzer was not final before ?

public final class ClassicAnalyzer extends StopwordAnalyzerBase
>


--
Kumaran R

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org