You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by vempap <ph...@emc.com> on 2012/10/05 01:43:29 UTC

StandardTokenizer generation from JFlex grammar

Hello,

  I'm trying to generate the standard tokenizer again using the jflex
specification (StandardTokenizerImpl.jflex) but I'm not able to do so due to
some errors (I would like to create my own jflex file using the standard
tokenizer which is why I'm trying to first generate using that to get a hang
of things).

I'm using jflex 1.4.3 and I ran into the following error:

Error in file "<filename>" (line 64): 
Syntax error.
HangulEx       = (!(!\p{Script:Hangul}|!\p{WB:ALetter})) ({Format} |
{Extend})*


Also, I tried installing an eclipse plugin from
http://cup-lex-eclipse.sourceforge.net/ which I thought would provide
options similar to JavaCC (http://eclipse-javacc.sourceforge.net/) through
which we can generate classes within eclipse - but had a hard luck.

Any help would be very helpful.

Regards,
Phani.



--
View this message in context: http://lucene.472066.n3.nabble.com/StandardTokenizer-generation-from-JFlex-grammar-tp4011941.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: StandardTokenizer generation from JFlex grammar

Posted by vempap <ph...@emc.com>.

Thanks. I'll look into it.



--
View this message in context: http://lucene.472066.n3.nabble.com/StandardTokenizer-generation-from-JFlex-grammar-tp4011941p4012092.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: StandardTokenizer generation from JFlex grammar

Posted by Ahmet Arslan <io...@yahoo.com>.

> > I'm using jflex 1.4.3 and I ran into the following
> 
> You need to use trunk. There is info about this in *.flex
> file. jflex-1.5.0-SNAPSHOT.jar
> 

Taken from ClassicTokenizerImpl.jflex :

"WARNING: if you change ClassicTokenizerImpl.jflex and need to regenerate the tokenizer, only use the trunk version of JFlex 1.5 at the moment!"

Re: StandardTokenizer generation from JFlex grammar

Posted by Ahmet Arslan <io...@yahoo.com>.

>   I'm trying to generate the standard tokenizer again
> using the jflex
> specification (StandardTokenizerImpl.jflex) but I'm not able
> to do so due to
> some errors (I would like to create my own jflex file using
> the standard
> tokenizer which is why I'm trying to first generate using
> that to get a hang
> of things).
> 
> I'm using jflex 1.4.3 and I ran into the following error:

You need to use trunk. There is info about this in *.flex file. jflex-1.5.0-SNAPSHOT.jar