You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@netbeans.apache.org by "matthiasblaesing (via GitHub)" <gi...@apache.org> on 2023/06/18 15:05:37 UTC
[GitHub] [netbeans] matthiasblaesing opened a new pull request, #6091: Rust lexer has to be able to handle singular quotes '/"
matthiasblaesing opened a new pull request, #6091:
URL: https://github.com/apache/netbeans/pull/6091
In a valid Rust programm quotes occur in pairs to form character, string or byte string literals. However while inputting the programm the lexer must still be able to handle them. The lexer has to produce a token in that case so that the NetBeans infrastructure relying on the token stream can work.
It is not a problem, that the resulting token stream can't be parsed.
Concrete cases:
Removing parts of single quote structure. The pipe symbols are not part of the code. The area from the first pipe to the second pipe is selected and removed.
```
fn main() {
println!('|x');
}
|
```
Enter single quote ('). The pipe symbol is not part of the code, at that position a single quote is entered.
```
fn main() {
println!(|
```
This partially reverts b62889b5c2799cd58c925e8fd560088d34947418 and introduces a different fix, that handles the "lonely" characters as explicit tokens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] matthiasblaesing commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "matthiasblaesing (via GitHub)" <gi...@apache.org>.
matthiasblaesing commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1606211464
Thank you all for review. I agree, that the lexer problem with characters outside the BMP should be tackled outside this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] matthiasblaesing commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "matthiasblaesing (via GitHub)" <gi...@apache.org>.
matthiasblaesing commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1605564798
Thank you for the test file. This is indeed a worst-case. I think, that we are seeing a problem in the NetBeans lexer infrastructure when characters outside the basic Unicode Plane are encountered.
One problematic character in that file is the code point 66349. That code point is:
https://www.compart.com/de/unicode/U+1032D
In java we normally deal with `char`s or character arrays. The 16bit size of char limits it to the BMP. For that reason Java encodes codepoints outside of the BMP as surrogate pairs (https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF).
It seems, that at least one problem is, that the antlr lexer does not expect surrogate pairs and so we would need to recombine first and feed the resulting int into the lexer. A quick test looks promising. However the rendering is still off. I'll look further into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] vieiro commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "vieiro (via GitHub)" <gi...@apache.org>.
vieiro commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1605608048
I'll take a look at how ANTLR4 Support handles exceptions as time permits. We may want to return some sort of "error token" or similar in ANTLR4Support on these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] matthiasblaesing commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "matthiasblaesing (via GitHub)" <gi...@apache.org>.
matthiasblaesing commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1596236545
Totally not a Rust expert here, but to me the specification implies, that `'x` is a lifetime or loop label, while `'` is not. The lexer has to split the cases:
- `'` - a lonely single quote
- `'dummy` - a singlequote followed by characters, that can be used as identifiers
- `'d'` - a single quote followed by a character, followed by a single quote
This is decideable with a look ahead of 2. This should be trivial for the code generation of antlr.
While reading your reply I realise, that the currently used grammar does not map lifetimes at all. That needs to be handled later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] BradWalker commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "BradWalker (via GitHub)" <gi...@apache.org>.
BradWalker commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1596233030
I could use some clarification here. I thought that single quote, ', deals with Rust lifetime rules..
https://doc.rust-lang.org/nomicon/lifetimes.html
Does this agree with your interpretation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] vieiro commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "vieiro (via GitHub)" <gi...@apache.org>.
vieiro commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1599303986
I'll need a few days to look at this one, I'm afraid...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] vieiro commented on pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "vieiro (via GitHub)" <gi...@apache.org>.
vieiro commented on PR #6091:
URL: https://github.com/apache/netbeans/pull/6091#issuecomment-1605307140
As a quick test, I cloned and opened the regex project (https://github.com/rust-lang/regex) and browsed some files.
@matthiasblaesing , when opening this file: https://github.com/rust-lang/regex/blob/master/regex-syntax/src/unicode_tables/age.rs the lexer reported a syntax error, and the whole editor system broke (the SwingEDT behaves badly after that).
This happens in master as well. The Lexer reports an error, and this is catastrophic for the ANTLR4 Lexer Support module (the editor freezes).
We may want to investigate what's going on in ANTLR4 Lexer Support (how to handle failures in the antlr4 lexers). Can't tell if we want to merge this one before or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] matthiasblaesing merged pull request #6091: Rust lexer has to be able to handle singular quotes '/"
Posted by "matthiasblaesing (via GitHub)" <gi...@apache.org>.
matthiasblaesing merged PR #6091:
URL: https://github.com/apache/netbeans/pull/6091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists