Re: [PATCH] for unicode problem over 0xff characters

["Daniel F. Savarese" <df...@savarese.org>]

>I made a patch for unicode problem at Perl5Compiler.java and
>Perl5Matcher.java.

Thanks very much for your efforts and providing a patch.  Unfortunately,
the fundamental approach is not the ultimate one that should be taken,
so I don't think we should apply the patch or some variation thereof.  The
problem is that it can cause excessive memory use (e.g., up to 8K per
character class) since it follows the same bitfield approach used for
the 8-bit ASCII character classes.  Handling 16-bit characters in a
character class requires a different approach, which is a little less
efficient in matching time, but much more efficient in the use of memory.
Implementing the "proper" solution, however, will require a good investment
of time and some significant code changes.

However, as a stopgap measure, we could implement the bitfield approach,
making it clear in comments and in the CHANGES file that it is temporary.
We could take an informal vote to that effect.  The problem with the patch
you posted is that it allows for incorrect matches or
ArrayOutOfBoundsExceptions to be thrown if the input contains characters
outside of the upper limit of the range of the character class range:

--- 806,812 ----
   if(nextChar == __EOS && inputRemains)
     nextChar = __input[input];

!  if((__program[current + (nextChar >> 4)] &
       (1 << (nextChar & 0xf))) != 0)
     return false;

It also doesn't make the necessary changes to Perl5Debug, which would
break after this patch was applied.  To generalize the current bitfield
implementation, you need to store the ultimate size of the bitfield and
make a comparision to ensure the input character can be used to index
into the bitfield.  At any rate, making these changes is rather
straightforward; I never did because of the desire to conserve memory.
So the question is, do people feel we should implement a temporary
stopgap measure, or just wait to "do it right"?

daniel

Re: [PATCH] for unicode problem over 0xff characters

[Takashi Okamoto <to...@rd.nttdata.co.jp>]

Thank you, daniel.

> The problem is that it can cause excessive memory use (e.g., up to
> 8K per character class) since it follows the same bitfield approach
> used for the 8-bit ASCII character classes.
  
I agree my solution is poor(;_;
But 8k memory is garbage for recent high performance PC:-)

> ArrayOutOfBoundsExceptions to be thrown if the input contains
> characters outside of the upper limit of the range of the character
> class range:

> !  if((__program[current + (nextChar >> 4)] &

I mistook.

 __program = new char[Character.MAX_VALUE >> 4];

This isn't  enough allocation for upper limit. 
Limit of __program depend on both 'char >> 4' and 'current'.
How many limit is it?

> It also doesn't make the necessary changes to Perl5Debug, which
> would break after this patch was applied.  

Excuse me , I have never used Perl5Debug. 
Is it fatal error? 
If you have your idea , could you tell me your idea ?

> So the question is, do people feel we should implement a temporary
> stopgap measure, or just wait to "do it right"?

How long will we wait for "proper" solution?
If I need many time for 'proper' release , I want temporary.


Jakarta-regexp ,gnu-regexp and xerces-j(IBM Regexp4j) didn't have this
problem. 
Not english people may run away to their library.
ORO have more function . This problem is just one disappointment for us.

Regards.
-------------
Takashi Okamoto