You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Joey Lawrance <la...@cs.orst.edu> on 2004/08/23 13:46:13 UTC
CJK Support for HTMLParser.jj
So, I needed the ability to parse Japanese HTML documents using
lucene-ja for my job. I was frustrated when I got HTML parser errors on
valid Japanese HTML. I digged a little, and I was excited to see the
StandardTokenizer.jj grammar already had CJK ranges defined in it. I
copied/pasted the CJK ranges from StandardTokenizer.jj into
HTMLParser.jj and added CJK as a type of token and viola! I can now
parse Japanese HTML documents using lucene-ja. Believe me, lucene-ja is
very crippled without this ability!
I've attached the HTMLParser.jj file that successfully parses Japanese
HTML for indexing. It is derived from the lucene-1.4-final version of
HTMLParser.jj, and I've attached a patch (against lucene-1.4-final).
Obviously, I don't have CVS commit access (and I'm not requesting it),
but I'd like to contribute this patch back to Lucene as it has been
absolutely invaluable for my work, and this is my way of saying "thank
you!" Let me know if a patch against CVS would be more convenient, or if
this patch is even worthy of being included in Lucene. I certainly think
it is. :-)
Joey
Re: CJK Support for HTMLParser.jj
Posted by Joey Lawrance <la...@cs.orst.edu>.
I got the same warning when I compiled the patch. I haven't tried my
patch with the patch for Bug 30844 (or the latest CVS) to see if it
removes the warning. I assume that would fix the problem, but I haven't
tested that theory out. I'll get around to that after I finish my
current work (which uses Lucene to index Japanese documents) under a
looming deadline. :-)
Joey
On Tuesday, September 7, 2004, at 01:19 PM, Daniel Naber wrote:
> On Monday 23 August 2004 13:46, Joey Lawrance wrote:
>
>> I've attached the HTMLParser.jj file that successfully parses Japanese
>> HTML for indexing.
>
> Joey,
>
> thanks for the patch. When I compile it with "ant javacc-HTMLParser" I
> get
> this warning:
>
> "Warning: Line 364, Column 3: Non-ASCII characters used in regular
> expression.
> Please make sure you use the correct Reader when you create the parser
> that
> can handle your character set."
>
> Is it okay to get this warning? The line the warning refers to is this
> one:
>
> | < CJK: // non-alphabets
>
> Besides that, the patch seems to work, i.e. the parser doesn't stop on
> Japanese HTML files anymore, but that's all I can say, as I don't speak
> Japanese.
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CJK Support for HTMLParser.jj
Posted by Daniel Naber <da...@t-online.de>.
On Monday 23 August 2004 13:46, Joey Lawrance wrote:
> I've attached the HTMLParser.jj file that successfully parses Japanese
> HTML for indexing.
Joey,
thanks for the patch. When I compile it with "ant javacc-HTMLParser" I get
this warning:
"Warning: Line 364, Column 3: Non-ASCII characters used in regular
expression.
Please make sure you use the correct Reader when you create the parser that
can handle your character set."
Is it okay to get this warning? The line the warning refers to is this one:
| < CJK: // non-alphabets
Besides that, the patch seems to work, i.e. the parser doesn't stop on
Japanese HTML files anymore, but that's all I can say, as I don't speak
Japanese.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org