You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@jena.apache.org by GitBox <gi...@apache.org> on 2022/09/22 04:48:06 UTC
[GitHub] [jena] bvosburgh-tq opened a new pull request, #1552: GH-1551: fix Turtle parser handling of tabs in URIs
bvosburgh-tq opened a new pull request, #1552:
URL: https://github.com/apache/jena/pull/1552
GitHub issue resolved #1551
Pull request Description:
See summary.
----
- [x] Tests are included.
- [n/a] Documentation change and updates are provided for the [Apache Jena website](https://github.com/apache/jena-site/)
- [x] Commits have been squashed to remove intermediate development commit messages.
- [x] Key commit messages start with the issue number (GH-xxxx or JENA-xxxx)
By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the [Contributor's Agreement](https://www.apache.org/licenses/contributor-agreements.html).
----
See the [Apache Jena "Contributing" guide](https://github.com/apache/jena/blob/main/CONTRIBUTING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org
[GitHub] [jena] afs closed pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
Posted by GitBox <gi...@apache.org>.
afs closed pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
URL: https://github.com/apache/jena/pull/1552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org
[GitHub] [jena] afs commented on a diff in pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
Posted by GitBox <gi...@apache.org>.
afs commented on code in PR #1552:
URL: https://github.com/apache/jena/pull/1552#discussion_r977427029
##########
jena-arq/src/main/java/org/apache/jena/riot/tokens/TokenizerText.java:
##########
@@ -527,7 +527,7 @@ private String readIRI() {
// Probably a corrupt file so treat as fatal.
fatal("Bad character in IRI (bad character: '<'): <%s[<]...>", stringBuilder.toString()); return null;
case TAB:
- error("Bad character in IRI (Tab character): <%s[tab]...>", stringBuilder.toString()); return null;
+ error("Bad character in IRI (Tab character): <%s[tab]...>", stringBuilder.toString()); break;
Review Comment:
Looks good.
##########
jena-arq/src/test/java/org/apache/jena/riot/lang/TestTurtleTerms.java:
##########
@@ -237,6 +241,41 @@
@Test public void turtle_152() { parse("[ a <y> ; a <z> ] . ") ; }
@Test public void turtle_153() { parse("[ a <z>, <z1> ] . ") ; }
+ // fatal characters
+ @Test(expected=RiotException.class)
+ public void turtle_154() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_<_open_angle_bracket> . "); }
+ @Test(expected=RiotException.class)
+ public void turtle_155() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_>_close_angle_bracket> . "); }
+ @Test(expected=RiotException.class)
+ public void turtle_156() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\\_backslash> . "); }
+ @Test(expected=RiotException.class)
+ public void turtle_157() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\n_newline> . "); }
+ @Test(expected=RiotException.class)
+ public void turtle_158() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\r_carriage_return> . "); }
+
+ // warning characters
+ @Test public void turtle_159() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\t_tab> . "); }
+ @Test public void turtle_160() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_ _space> . "); }
+ @Test public void turtle_161() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_{_open_curly_brace> . "); }
+ @Test public void turtle_162() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_}_close_curly_brace> . "); }
+ @Test public void turtle_163() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\"_double_quote> . "); }
+ @Test public void turtle_164() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_|_pipe> . "); }
+ @Test public void turtle_165() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_^_caret> . "); }
+ @Test public void turtle_166() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_`_backtick> . "); }
+ @Test public void turtle_167() { laxParse("<http://example/foo> <http://example/bar> <http://example/invalid/iri/with_\f_form_feede> . "); }
+
+ private static void laxParse(String string) {
+ PeekReader r = PeekReader.readString(string) ;
+ Tokenizer tokenizer = TokenizerText.create()
Review Comment:
Simpler: `TokenizerText.create().fromString(string)` - no need to create the `PeekReader`.
(Some or all of `TestTokenizer` predates `fromString`.)
##########
jena-arq/src/test/java/org/apache/jena/riot/lang/TestTurtleTerms.java:
##########
@@ -237,6 +241,41 @@
@Test public void turtle_152() { parse("[ a <y> ; a <z> ] . ") ; }
@Test public void turtle_153() { parse("[ a <z>, <z1> ] . ") ; }
+ // fatal characters
Review Comment:
These are Turtle tests and with the layers over the tokenizer, the tests can't tell where RiotException is coming from. e.g. `turtle_155` - the URI is legal, it sends at `...with_>` leaving junk and error "Triples not terminated by DOT" from the Turtle parser.
The tokenizer tests are in `TestTokenizer`. There is a function `testExpectWarning` that handles testing for warnings.
If you want to leave Turtle behavior tests in, then fine - but the tokenizer is used in various places so it needs its own tests for behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org
[GitHub] [jena] bvosburgh-tq commented on a diff in pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
Posted by GitBox <gi...@apache.org>.
bvosburgh-tq commented on code in PR #1552:
URL: https://github.com/apache/jena/pull/1552#discussion_r979138722
##########
jena-arq/src/test/java/org/apache/jena/riot/lang/TestTurtleTerms.java:
##########
@@ -237,6 +241,41 @@
@Test public void turtle_152() { parse("[ a <y> ; a <z> ] . ") ; }
@Test public void turtle_153() { parse("[ a <z>, <z1> ] . ") ; }
+ // fatal characters
Review Comment:
> The tokenizer tests are in `TestTokenizer` etc.
Thanks for the feedback. I started with Turtle parsing tests because that was how the problem manifested itself to me. I have removed these tests and put the single relevant test in `TestTokenizer`, where it belongs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org
[GitHub] [jena] afs commented on a diff in pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
Posted by GitBox <gi...@apache.org>.
afs commented on code in PR #1552:
URL: https://github.com/apache/jena/pull/1552#discussion_r979224567
##########
jena-arq/src/test/java/org/apache/jena/riot/tokens/TestTokenizer.java:
##########
@@ -1044,21 +1045,38 @@ public void tokenStr_replacmentChar_blankNode_1() {
// and no escaped characters for blank node labels.
}
+ @Test
Review Comment:
Other tests use `@Test(expected = RiotParseException.class)`.
Errors are expected to be exceptions for `TokenizerText`.
Does this need to be different?
Why not the pattern of other tests:
```
@Test(expected=RiotException.class)
public void tokenIri_tab() {
Tokenizer tokenizer = tokenizer("<http://example/invalid/iri/with_\t_tab>") ;
testNextToken(tokenizer, TokenType.IRI) ;
}
```
##########
jena-arq/src/test/java/org/apache/jena/riot/tokens/TestTokenizer.java:
##########
@@ -1044,21 +1045,38 @@ public void tokenStr_replacmentChar_blankNode_1() {
// and no escaped characters for blank node labels.
}
+ @Test
+ public void tokenIri_tab() {
+ testExpectError("<http://example/invalid/iri/with_\t_tab>", TokenType.IRI, 1);
+ }
+
private static Token testExpectWarning(String input, TokenType expectedTokenType, int warningCount) {
- PeekReader r = PeekReader.readString(input);
- return testExpectWarning(r, expectedTokenType, warningCount);
+ return testExpectWarningOrError(input, expectedTokenType, warningCount, 0);
}
private static Token testExpectWarning(PeekReader r, TokenType expectedTokenType, int warningCount) {
- ErrorHandlerRecorder errHandler = new ErrorHandlerRecorder(errorHandlerSimple());
Review Comment:
`errorHandlerSimple` throws exceptions. This is intentional.
`errorHandlerWarning` does not.
This change makes a difference to all tests using `testExpectWarning`.
If you want to propose such a general change, then that is a different issue. It isn't limited to tabs in URIs.
Testing what happens if the parser continues is not part of this test suite (and in general is not defined).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org
[GitHub] [jena] afs commented on pull request #1552: GH-1551: fix Turtle parser handling of tabs in URIs
Posted by GitBox <gi...@apache.org>.
afs commented on PR #1552:
URL: https://github.com/apache/jena/pull/1552#issuecomment-1264607340
Done with #1571.
@bvosburgh-tq -- please check it now does not cause an NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org