You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ph...@mailbox.org on 2017/03/17 20:05:40 UTC

How to customize the delimiters used by the WordDelimiterFilter in Lucene?

Hi,

I am trying to index words like 'e-mail' as 'email', 'e mail' and 'e-mail' with Lucene 4.4.0.

Lucene's WordDelimiterFilter should be ideal for this. However, it treats every(?) non-alphanumeric character as a delimiter. So, terms like 'C++' are transformed to 'C', which is not what I want.

Apparently, Solr allows to specify custom delimiters. But how can I do it in Lucene?

I have looked into the documentation and the 'byte[] charTypeTable' parameter in the Constructor looked promising. But it seems to have no effect if I specify some delimiters in a charTypeTable.

Thank you!

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: How to customize the delimiters used by the WordDelimiterFilter in Lucene?

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,

May be look at the factory class to see how types argument is handled?

Ahmet


On Friday, March 17, 2017 11:05 PM, "phauly@mailbox.org" <ph...@mailbox.org> wrote:



Hi,


I am trying to index words like 'e-mail' as 'email', 'e mail' and 'e-mail' with Lucene 4.4.0.


Lucene's WordDelimiterFilter should be ideal for this. However, it treats every(?) non-alphanumeric character as a delimiter. So, terms like 'C++' are transformed to 'C', which is not what I want.


Apparently, Solr allows to specify custom delimiters. But how can I do it in Lucene?


I have looked into the documentation and the 'byte[] charTypeTable' parameter in the Constructor looked promising. But it seems to have no effect if I specify some delimiters in a charTypeTable.


Thank you!


---------------------------------------------------------------------

To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org

For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org