You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by OBender <os...@hotmail.com> on 2009/07/17 20:29:25 UTC
Tokenizer queston: how can I force ? and ! to be separate tokens?
Hi All,
I need to make ? and ! characters to be a separate token e.g. to split [how
are you?] in to 4 tokens [how], [are], [you] and [?] what would be the best
way to do this?
Thanks
RE: Tokenizer question: how can I force ? and ! to be separate tokens?
Posted by OBender <os...@hotmail.com>.
Thanks, I think I got it.
-----Original Message-----
From: John Byrne [mailto:john.byrne@propylon.com]
Sent: Friday, July 17, 2009 2:43 PM
To: java-user@lucene.apache.org
Subject: Re: Tokenizer queston: how can I force ? and ! to be separate
tokens?
Yes, you could even use the WhitespaceTokenizer and then look for the
symbols in a token filter. You would get [you?] as a single token; your
job in the token filter is then to store the [?] and return the [you].
The next time the token filter is called for the next token, you return
the [?] that you stored previously.
If you're already using something that's grammar-based (such as
StandardTokenizer) then you could add the "?" to the grammar as a
separate token. If you can figure out how to do this from looking at the
grammar file, then it's probably the simplest way.
-John
Matthew Hall wrote:
> I'd think extending WhiteSpaceTokenizer would be a good place to start.
>
> Then create a new Analyzer that exactly mirrors your current Analyzer,
> with the exception that it uses your new tokenizer instead of
> WhiteSpaceTokenizer (Well.. there is of course my assumption that you
> are using an Analyzer that already uses WhiteSpaceTokenizer... but you
> likely are)
>
> OBender wrote:
>> Hi All,
>>
>>
>>
>> I need to make ? and ! characters to be a separate token e.g. to
>> split [how
>> are you?] in to 4 tokens [how], [are], [you] and [?] what would be
>> the best
>> way to do this?
>>
>>
>>
>> Thanks
>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.392 / Virus Database: 270.13.18/2243 - Release Date: 07/17/09
06:08:00
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Why next(Token) in CharTokenizer is final?
Posted by Uwe Schindler <uw...@thetaphi.de>.
In general all TokenStream classes from Lucene core are/should be final,
because should simply put an additional TokenFilter into the chain to modify
the tokenization.
A few of them are not final (CharTokenizer is abstract), but here the next()
methods are final because of the above explained reason, because you should
only be able to customize the behaviour (which chars are whitespace and how
they are "normalized"), not the implementation of the TokenStream (which
could break a lot of things, see below).
If you want to plug in additional custom filtering into the TokenStream
chain, create a new TokenFilter and plug it ontop of CharTokenizer
subclasses, do not modify the core classes!
And this is now very important for the transition to the new TokenStream
API, see https://issues.apache.org/jira/browse/LUCENE-1693
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: OBender [mailto:osya_bender@hotmail.com]
> Sent: Friday, July 17, 2009 10:47 PM
> To: java-user@lucene.apache.org
> Subject: Why next(Token) in CharTokenizer is final?
>
> Hi All,
>
> I think this is a question to Lucene dev team.
> Why the next(Token) method of CharTokenizer was made final?
> It is quite inconvenient and I don't see the reason why it is so.
>
> Thanks.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Why next(Token) in CharTokenizer is final?
Posted by OBender <os...@hotmail.com>.
Hi All,
I think this is a question to Lucene dev team.
Why the next(Token) method of CharTokenizer was made final?
It is quite inconvenient and I don't see the reason why it is so.
Thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Tokenizer queston: how can I force ? and ! to be separate tokens?
Posted by John Byrne <jo...@propylon.com>.
Yes, you could even use the WhitespaceTokenizer and then look for the
symbols in a token filter. You would get [you?] as a single token; your
job in the token filter is then to store the [?] and return the [you].
The next time the token filter is called for the next token, you return
the [?] that you stored previously.
If you're already using something that's grammar-based (such as
StandardTokenizer) then you could add the "?" to the grammar as a
separate token. If you can figure out how to do this from looking at the
grammar file, then it's probably the simplest way.
-John
Matthew Hall wrote:
> I'd think extending WhiteSpaceTokenizer would be a good place to start.
>
> Then create a new Analyzer that exactly mirrors your current Analyzer,
> with the exception that it uses your new tokenizer instead of
> WhiteSpaceTokenizer (Well.. there is of course my assumption that you
> are using an Analyzer that already uses WhiteSpaceTokenizer... but you
> likely are)
>
> OBender wrote:
>> Hi All,
>>
>>
>>
>> I need to make ? and ! characters to be a separate token e.g. to
>> split [how
>> are you?] in to 4 tokens [how], [are], [you] and [?] what would be
>> the best
>> way to do this?
>>
>>
>>
>> Thanks
>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.392 / Virus Database: 270.13.18/2243 - Release Date: 07/17/09 06:08:00
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Tokenizer queston: how can I force ? and ! to be separate tokens?
Posted by Matthew Hall <mh...@informatics.jax.org>.
I'd think extending WhiteSpaceTokenizer would be a good place to start.
Then create a new Analyzer that exactly mirrors your current Analyzer,
with the exception that it uses your new tokenizer instead of
WhiteSpaceTokenizer (Well.. there is of course my assumption that you
are using an Analyzer that already uses WhiteSpaceTokenizer... but you
likely are)
OBender wrote:
> Hi All,
>
>
>
> I need to make ? and ! characters to be a separate token e.g. to split [how
> are you?] in to 4 tokens [how], [are], [you] and [?] what would be the best
> way to do this?
>
>
>
> Thanks
>
>
>
--
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall@informatics.jax.org
(207) 288-6012
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org