Dictionary support in tokenizer and sentence detector

[Jörn Kottmann <ko...@gmail.com>]

Hello,

the dictionary support in these two components should
take proper care for the case sensitivity.

Currently it is not well defined, because a String object
in java is only equals if they match with the exact case.
So if a user prepares a case insensitive dictionary, the matching
would still be case sensitive, which is confusing.

I suggest that we use the Dictionary object directly for the look up,
and remove Dictionary.asStringSet.

Any opinions on how to handle this?

Jörn

Re: Dictionary support in tokenizer and sentence detector

[Jörn Kottmann <ko...@gmail.com>]

On 8/23/11 3:30 PM, william.colen@gmail.com wrote:
> Why? If the user prepares a Dictionary and convert it to a Set using the
> asStringSet he gets a special Set. This implementation includes a contains
> method that as I can see should handle the case. I exercised a little more
> with DictionaryAsSetCaseInsensitiveTest and DictionaryAsSetCaseSensitiveTest
> and it looks OK to me.

I was not aware of this, sorry.

+1 to keep it as it is now.

Jörn

Re: Dictionary support in tokenizer and sentence detector

["william.colen@gmail.com" <wi...@gmail.com>]

On Tue, Aug 23, 2011 at 5:46 AM, Jörn Kottmann <ko...@gmail.com> wrote:

> Hello,
>
> the dictionary support in these two components should
> take proper care for the case sensitivity.
>
> Currently it is not well defined, because a String object
> in java is only equals if they match with the exact case.
> So if a user prepares a case insensitive dictionary, the matching
> would still be case sensitive, which is confusing.
>
>
Why? If the user prepares a Dictionary and convert it to a Set using the
asStringSet he gets a special Set. This implementation includes a contains
method that as I can see should handle the case. I exercised a little more
with DictionaryAsSetCaseInsensitiveTest and DictionaryAsSetCaseSensitiveTest
and it looks OK to me.

I suggest that we use the Dictionary object directly for the look up,
> and remove Dictionary.asStringSet.
>
> Any opinions on how to handle this?
>

+1 to use it directly, but how to do it backward compatible?