You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@netbeans.apache.org by GitBox <gi...@apache.org> on 2023/01/05 07:43:52 UTC

[GitHub] [netbeans] lkishalmi opened a new pull request, #5206: Initial implementation of ANTLR4 Lexer Support

lkishalmi opened a new pull request, #5206:
URL: https://github.com/apache/netbeans/pull/5206

   
   Take number 2.
   
   This is a bunch of collected knowledge, help to integrate ANTLR v4 based Lexers into NetBeans. This PR does not include the change of the existing ANTLR v4 based languages. If this one goes through, I'd provide separate PR-s for:
    - ANTLR Language Support
    - TOML Support
    - JSON Support
   
   I could use this on my upcoming projects for:
    - Terraform, for two or more lexers
    - GoLang Lexer
    - Maybe: HTTP Lexer, may create a new Language Support tutorial featuring that simple grammar.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] matthiasblaesing commented on pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
matthiasblaesing commented on PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#issuecomment-1375911582

   I know this is already in, but one thing I thought about when looking at this was: are we prepared to make this API yet or should this be a purely internal package (i.e. Friend-only API without external users)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] lkishalmi commented on pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
lkishalmi commented on PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#issuecomment-1375999133

   Well, this one does not introduce something big/groundbreaking. It's only 4 classes. Two of them: `AbstractAntlrLexerBridge` and `LexerInputCharStream` are actually implementations of an existing API. The `AbstractAntlrLexerBridge.LexerState` duties and usage is strict and well defined, that leaves `AntlrTokenSequence` as new here. That one is also simple and straight forward enough.
   
   I could imagine that there could be bugs in the implementation, though I'd not expect the API change without keeping backward compatibility any time soon. (Unless ANTLR forces us to do so...)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] lkishalmi commented on pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
lkishalmi commented on PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#issuecomment-1376003567

   Hmm. Could make the `LexerInputCharStream` package private though...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] lkishalmi commented on a diff in pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
lkishalmi commented on code in PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#discussion_r1064222808


##########
ide/lexer.antlr4/test/unit/src/org/netbeans/spi/lexer/antlr4/AntlrTokenSequenceTest.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.netbeans.spi.lexer.antlr4;
+
+import org.antlr.grammars.dummy.DummyLexer;
+import org.antlr.v4.runtime.CharStreams;
+import org.junit.Test;
+import static org.junit.Assert.*;
+import static org.netbeans.spi.lexer.antlr4.AntlrTokenSequence.DEFAULT_CHANNEL;
+
+/**
+ *
+ * @author Laszlo Kishalmi
+ */
+public class AntlrTokenSequenceTest {
+
+    public AntlrTokenSequenceTest() {
+    }
+
+    /**
+     * Test of seekTo method, of class AntlrTokenSequence.
+     */
+    @Test
+    public void testSeekTo1() {
+        System.out.println("seekTo");

Review Comment:
   This one is generated by NetBeans. I can remove them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] lkishalmi merged pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
lkishalmi merged PR #5206:
URL: https://github.com/apache/netbeans/pull/5206


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] mbien commented on a diff in pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
mbien commented on code in PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#discussion_r1062984373


##########
ide/lexer.antlr4/test/unit/src/org/netbeans/spi/lexer/antlr4/AntlrTokenSequenceTest.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.netbeans.spi.lexer.antlr4;
+
+import org.antlr.grammars.dummy.DummyLexer;
+import org.antlr.v4.runtime.CharStreams;
+import org.junit.Test;
+import static org.junit.Assert.*;
+import static org.netbeans.spi.lexer.antlr4.AntlrTokenSequence.DEFAULT_CHANNEL;
+
+/**
+ *
+ * @author Laszlo Kishalmi
+ */
+public class AntlrTokenSequenceTest {
+
+    public AntlrTokenSequenceTest() {
+    }
+
+    /**
+     * Test of seekTo method, of class AntlrTokenSequence.
+     */
+    @Test
+    public void testSeekTo1() {
+        System.out.println("seekTo");

Review Comment:
   are the printlns needed here? Consider printing more info on assert failure but less when its green.



##########
ide/lexer.antlr4/test/unit/src/org/antlr/grammars/dummy/DummyLexer.java:
##########
@@ -0,0 +1,175 @@
+// Generated from java-escape by ANTLR 4.11.1
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// DO NOT EDIT THIS FILE MANUALLY!
+// SEE build.xml FOR INSTRUCTIONS
+package org.antlr.grammars.dummy;
+
+
+import org.antlr.v4.runtime.Lexer;
+import org.antlr.v4.runtime.CharStream;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.TokenStream;
+import org.antlr.v4.runtime.*;
+import org.antlr.v4.runtime.atn.*;
+import org.antlr.v4.runtime.dfa.DFA;
+import org.antlr.v4.runtime.misc.*;
+
+@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast", "CheckReturnValue"})
+public class DummyLexer extends Lexer {
+	static { RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION); }
+
+	protected static final DFA[] _decisionToDFA;
+	protected static final PredictionContextCache _sharedContextCache =
+		new PredictionContextCache();
+	public static final int
+		BLOCK_COMMENT=1, WS=2, LEXER=3, ID=4, ERROR=5;
+	public static final int
+		COMMENT=2;
+	public static String[] channelNames = {
+		"DEFAULT_TOKEN_CHANNEL", "HIDDEN", "COMMENT"
+	};
+
+	public static String[] modeNames = {
+		"DEFAULT_MODE"
+	};
+
+	private static String[] makeRuleNames() {
+		return new String[] {
+			"BLOCK_COMMENT", "WS", "LEXER", "ID", "ERROR"
+		};
+	}
+	public static final String[] ruleNames = makeRuleNames();
+
+	private static String[] makeLiteralNames() {
+		return new String[] {
+			null, null, null, "'lexer'", null, "'.'"
+		};
+	}
+	private static final String[] _LITERAL_NAMES = makeLiteralNames();
+	private static String[] makeSymbolicNames() {
+		return new String[] {
+			null, "BLOCK_COMMENT", "WS", "LEXER", "ID", "ERROR"
+		};
+	}
+	private static final String[] _SYMBOLIC_NAMES = makeSymbolicNames();
+	public static final Vocabulary VOCABULARY = new VocabularyImpl(_LITERAL_NAMES, _SYMBOLIC_NAMES);
+
+	/**
+	 * @deprecated Use {@link #VOCABULARY} instead.
+	 */
+	@Deprecated
+	public static final String[] tokenNames;
+	static {
+		tokenNames = new String[_SYMBOLIC_NAMES.length];
+		for (int i = 0; i < tokenNames.length; i++) {
+			tokenNames[i] = VOCABULARY.getLiteralName(i);
+			if (tokenNames[i] == null) {
+				tokenNames[i] = VOCABULARY.getSymbolicName(i);
+			}
+
+			if (tokenNames[i] == null) {
+				tokenNames[i] = "<INVALID>";
+			}
+		}
+	}
+
+	@Override
+	@Deprecated
+	public String[] getTokenNames() {
+		return tokenNames;
+	}
+
+	@Override
+
+	public Vocabulary getVocabulary() {

Review Comment:
   free floating annotation :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

[GitHub] [netbeans] lkishalmi commented on a diff in pull request #5206: Initial implementation of ANTLR4 Lexer Support

Posted by GitBox <gi...@apache.org>.
lkishalmi commented on code in PR #5206:
URL: https://github.com/apache/netbeans/pull/5206#discussion_r1064220759


##########
ide/lexer.antlr4/test/unit/src/org/antlr/grammars/dummy/DummyLexer.java:
##########
@@ -0,0 +1,175 @@
+// Generated from java-escape by ANTLR 4.11.1
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// DO NOT EDIT THIS FILE MANUALLY!
+// SEE build.xml FOR INSTRUCTIONS
+package org.antlr.grammars.dummy;
+
+
+import org.antlr.v4.runtime.Lexer;
+import org.antlr.v4.runtime.CharStream;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.TokenStream;
+import org.antlr.v4.runtime.*;
+import org.antlr.v4.runtime.atn.*;
+import org.antlr.v4.runtime.dfa.DFA;
+import org.antlr.v4.runtime.misc.*;
+
+@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast", "CheckReturnValue"})
+public class DummyLexer extends Lexer {
+	static { RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION); }
+
+	protected static final DFA[] _decisionToDFA;
+	protected static final PredictionContextCache _sharedContextCache =
+		new PredictionContextCache();
+	public static final int
+		BLOCK_COMMENT=1, WS=2, LEXER=3, ID=4, ERROR=5;
+	public static final int
+		COMMENT=2;
+	public static String[] channelNames = {
+		"DEFAULT_TOKEN_CHANNEL", "HIDDEN", "COMMENT"
+	};
+
+	public static String[] modeNames = {
+		"DEFAULT_MODE"
+	};
+
+	private static String[] makeRuleNames() {
+		return new String[] {
+			"BLOCK_COMMENT", "WS", "LEXER", "ID", "ERROR"
+		};
+	}
+	public static final String[] ruleNames = makeRuleNames();
+
+	private static String[] makeLiteralNames() {
+		return new String[] {
+			null, null, null, "'lexer'", null, "'.'"
+		};
+	}
+	private static final String[] _LITERAL_NAMES = makeLiteralNames();
+	private static String[] makeSymbolicNames() {
+		return new String[] {
+			null, "BLOCK_COMMENT", "WS", "LEXER", "ID", "ERROR"
+		};
+	}
+	private static final String[] _SYMBOLIC_NAMES = makeSymbolicNames();
+	public static final Vocabulary VOCABULARY = new VocabularyImpl(_LITERAL_NAMES, _SYMBOLIC_NAMES);
+
+	/**
+	 * @deprecated Use {@link #VOCABULARY} instead.
+	 */
+	@Deprecated
+	public static final String[] tokenNames;
+	static {
+		tokenNames = new String[_SYMBOLIC_NAMES.length];
+		for (int i = 0; i < tokenNames.length; i++) {
+			tokenNames[i] = VOCABULARY.getLiteralName(i);
+			if (tokenNames[i] == null) {
+				tokenNames[i] = VOCABULARY.getSymbolicName(i);
+			}
+
+			if (tokenNames[i] == null) {
+				tokenNames[i] = "<INVALID>";
+			}
+		}
+	}
+
+	@Override
+	@Deprecated
+	public String[] getTokenNames() {
+		return tokenNames;
+	}
+
+	@Override
+
+	public Vocabulary getVocabulary() {

Review Comment:
   Well, that code is generated from ANTLR. I would not change that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists