[jira] [Comment Edited] (GROOVY-8662) support full unicode capabilities from antlr2 parser in antlr4

["Paul King (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GROOVY-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520986#comment-16520986 ] 

Paul King edited comment on GROOVY-8662 at 6/24/18 12:48 AM:
-------------------------------------------------------------

This doesn't impact supported unicode cases, e.g.:
{code:java}
def \u0061 = '\uuuuu0061'
def \u0061\u0062
def \u0061\u0062\u0063
def \u0061\u0062\u00639
def \u0061cC\u0062Bb\u00639aA
def a\u00615\u00626\u0063Z
def A\u00617\u00628\u0063z

def \u0061\u0062() {}

class \u0061 {
    def \u0061\u0062

    def \u0061cC\u0062Bb\u00639aA() {}
}
interface \u0061 {}
enum \u0061 {
    \u0061cC\u0062Bb\u00639aA, \u0061cC\u0062Bb\u00639aA2
}
trait \u0061 {}
@interface \u0061 {}
{code}
but some interesting implementation decisions need to be considered:
 * Should the pre-processor leave unicode characters (such as those shown above in this comment) alone if found to be in a place where the Groovy grammar can handle them? This would be to maximise correct node information.
 * Should steps be taken to try to preserve the original node positions? e.g. should zero-width spaces (U+200B) be inserted after expanded unicodes or the next space expanded with extra spaces (but not within strings). Again this is to preserve correct line/column information from the original source file.


was (Author: paulk):
This doesn't impact supported unicode cases, e.g.:
{code}
def \u0061 = '\uuuuu0061'
def \u0061\u0062
def \u0061\u0062\u0063
def \u0061\u0062\u00639
def \u0061cC\u0062Bb\u00639aA
def a\u00615\u00626\u0063Z
def A\u00617\u00628\u0063z

def \u0061\u0062() {}

class \u0061 {
    def \u0061\u0062

    def \u0061cC\u0062Bb\u00639aA() {}
}
interface \u0061 {}
enum \u0061 {
    \u0061cC\u0062Bb\u00639aA, \u0061cC\u0062Bb\u00639aA2
}
trait \u0061 {}
@interface \u0061 {}
{code}
but obviously, the pre-processor should leave such characters alone if it can to maximise correct node information.

> support full unicode capabilities from antlr2 parser in antlr4
> --------------------------------------------------------------
>
>                 Key: GROOVY-8662
>                 URL: https://issues.apache.org/jira/browse/GROOVY-8662
>             Project: Groovy
>          Issue Type: Improvement
>          Components: parser-antlr4
>            Reporter: Paul King
>            Priority: Minor
>
> The Antlr2 parser allows the entire source code to be entered as unicode characters, e.g. see
> {{groovy.EscapedUnicodeTest}}. For the Antlr4 parser, for efficiency reasons, this was limited to names, e.g. class names and variable names etc. The thinking at the time was that the full use of unicode was a very special use case that didn't warrant the performance hit.
> We could however, optionally run a pre-parser (enabled by some command-line switch perhaps). Doing so would cause inaccuracies however in node position information - since the pre-parser would alter the file during unicode processing.
> We regard such functionality as low priority but include this issue to document our thinking about the preferred approach if such functionality was introduced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)