You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by da...@comcast.net on 2003/09/26 16:39:34 UTC

StandardTokenizer CJK Support

In August of 2002, Che, Dong suggested a change to StandardTokenizer.jj that 
would supply some basic support for CJK. (msgNo:2164) A day later, Doug gave it 
+1. The suggested change was not added to CVS nor was there any further 
discussion on the mailing list. 

I'm working with an application in which certain fields are mixed language and 
this change is very useful. Is there a technical reason why this change was not 
made? 

Regards,

--Dan Rapp