You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by afs <gi...@git.apache.org> on 2016/10/27 09:27:47 UTC
[GitHub] jena pull request #184: JENA-1252: Tokenizer cleaup
GitHub user afs opened a pull request:
https://github.com/apache/jena/pull/184
JENA-1252: Tokenizer cleaup
Puts in place the structure for warnings not being tokenizer errors.
This is not enabled - it needs testing to see what knock-on effects it has. Ideally, it should be combined with parsers that can do some level of error recovery.
https://issues.apache.org/jira/browse/JENA-1252
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/afs/jena tokenizer-cleaup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/jena/pull/184.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #184
----
commit 4d130f509b18eea6136af2c8820977e0877e9733
Author: Andy Seaborne <an...@apache.org>
Date: 2016-10-26T19:42:18Z
Clean tokenizer; support for warnings (and continue)
commit 6b932a567cd829fc482d11139c7b6615c33bb48a
Author: Andy Seaborne <an...@apache.org>
Date: 2016-10-27T09:21:30Z
Reformat
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] jena pull request #184: JENA-1252: Tokenizer cleaup
Posted by afs <gi...@git.apache.org>.
Github user afs commented on a diff in the pull request:
https://github.com/apache/jena/pull/184#discussion_r85341210
--- Diff: jena-arq/src/main/java/org/apache/jena/riot/tokens/TokenizerText.java ---
@@ -60,6 +61,26 @@
private boolean finished = false ;
private TokenChecker checker = null ;
+ // The code assumes that errors throw exception and so stop parsing.
+ private ErrorHandler errorHandler = new ErrorHandler() {
--- End diff --
Will do. It is stateless though a stateless object is not very expensive to create!
The others are in ErrorHandlerFactory.
It maybe it becomes a parameter to the constructor.
Because letting dubious data through can lead to a lot of support load due to downstream database problems, this PR just adds mechanism and does not enable it. Hence it is keeping the default error handler in-file for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] jena pull request #184: JENA-1252: Tokenizer cleaup
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/jena/pull/184
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] jena pull request #184: JENA-1252: Tokenizer cleaup
Posted by ajs6f <gi...@git.apache.org>.
Github user ajs6f commented on a diff in the pull request:
https://github.com/apache/jena/pull/184#discussion_r85342117
--- Diff: jena-arq/src/main/java/org/apache/jena/riot/tokens/TokenizerText.java ---
@@ -60,6 +61,26 @@
private boolean finished = false ;
private TokenChecker checker = null ;
+ // The code assumes that errors throw exception and so stop parsing.
+ private ErrorHandler errorHandler = new ErrorHandler() {
--- End diff --
\U0001f44d to the scheme (not making it too easy to let cruft through).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] jena pull request #184: JENA-1252: Tokenizer cleaup
Posted by ajs6f <gi...@git.apache.org>.
Github user ajs6f commented on a diff in the pull request:
https://github.com/apache/jena/pull/184#discussion_r85338885
--- Diff: jena-arq/src/main/java/org/apache/jena/riot/tokens/TokenizerText.java ---
@@ -60,6 +61,26 @@
private boolean finished = false ;
private TokenChecker checker = null ;
+ // The code assumes that errors throw exception and so stop parsing.
+ private ErrorHandler errorHandler = new ErrorHandler() {
--- End diff --
I may be misunderstanding here, but couldn't this be `static`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---