You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Sebastian Hellmann <he...@informatik.uni-leipzig.de> on 2019/09/20 07:11:45 UTC
Riot warning for Unicode NFC IRIs
Hi all,
I really don't understand these RIOT warnings:
19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
<http://dbpedia.org/resource/Ranma_½> Code: 47/NOT_NFKC in PATH: The IRI
is not in Unicode Normal Form KC.
19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
<http://dbpedia.org/resource/Ranma_½> Code: 56/COMPATIBILITY_CHARACTER
in PATH: Bad character
for ½ NFC == NFD as well as NFKD == NFKC
Tested here: https://minaret.info/test/normalize.msp
Result string: ½
Result in hex: bd
Result string: 1⁄2
Result in hex: 31 2044 32
Both Wikipedia and DBpedia can not handle NFKC:
https://en.wikipedia.org/wiki/Ranma_1⁄2
Also the RDF Spec says to normalize to NFC:
https://www.w3.org/TR/rdf11-concepts/#section-IRIs
We are using these versions:
+- org.apache.jena:jena-core:jar:3.7.0:compile
[INFO] | | \- org.apache.jena:jena-base:jar:3.7.0:compile
Concretely I have these questions:
- can you check whether this is a bug or point me to the issue tracker,
where this was or should be recorded.
- How can I disable specifically these two warning messages, either by
configuration or by tuning log4j?
--
All the best,
Sebastian Hellmann
Director of Knowledge Integration and Linked Data Technologies (KILT)
Competence Center
at the Institute for Applied Informatics (InfAI) at Leipzig University
Executive Director of the DBpedia Association
Projects: http://dbpedia.org, http://nlp2rdf.org,
http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
<http://www.w3.org/community/ld4lt>
Homepage: http://aksw.org/SebastianHellmann
Research Group: http://aksw.org
Re: Riot warning for Unicode NFC IRIs
Posted by Sebastian Hellmann <he...@informatik.uni-leipzig.de>.
Is there a non-English user list for jena? IRIs have been around for
quite a while:
https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier
https://dbpedia.org/resource/Ranma_%C2%BD also works like
https://en.wikipedia.org/wiki/Ranma_%C2%BD
Both because they are the Percent-Encoded form of Unicode NFC.
The warning messages refers specifically to a different unicode normal
form and a compatibility character. And I don't know in what way this is
applicable to %C2%BD or ½ or why it should be NFKC in the first place.
-- Sebastian
On 20.09.19 09:48, Laura Morales wrote:
> Valid URIs are basically ASCII strings. I don't think you can stick any other character in there.
>
>
>> Sent: Friday, September 20, 2019 at 9:11 AM
>> From: "Sebastian Hellmann" <he...@informatik.uni-leipzig.de>
>> To: users@jena.apache.org
>> Subject: Riot warning for Unicode NFC IRIs
>>
>> Hi all,
>>
>> I really don't understand these RIOT warnings:
>>
>> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
>> <http://dbpedia.org/resource/Ranma_½> Code: 47/NOT_NFKC in PATH: The IRI
>> is not in Unicode Normal Form KC.
>> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
>> <http://dbpedia.org/resource/Ranma_½> Code: 56/COMPATIBILITY_CHARACTER
>> in PATH: Bad character
>>
>> for ½ NFC == NFD as well as NFKD == NFKC
>>
>> Tested here: https://minaret.info/test/normalize.msp
>>
>> Result string: ½
>> Result in hex: bd
>>
>> Result string: 1⁄2
>> Result in hex: 31 2044 32
>>
>> Both Wikipedia and DBpedia can not handle NFKC:
>> https://en.wikipedia.org/wiki/Ranma_1⁄2
>>
>> Also the RDF Spec says to normalize to NFC:
>> https://www.w3.org/TR/rdf11-concepts/#section-IRIs
>>
>>
>> We are using these versions:
>>
>> +- org.apache.jena:jena-core:jar:3.7.0:compile
>> [INFO] | | \- org.apache.jena:jena-base:jar:3.7.0:compile
>>
>> Concretely I have these questions:
>>
>> - can you check whether this is a bug or point me to the issue tracker,
>> where this was or should be recorded.
>>
>> - How can I disable specifically these two warning messages, either by
>> configuration or by tuning log4j?
>>
>>
>> --
>> All the best,
>> Sebastian Hellmann
>>
>> Director of Knowledge Integration and Linked Data Technologies (KILT)
>> Competence Center
>> at the Institute for Applied Informatics (InfAI) at Leipzig University
>> Executive Director of the DBpedia Association
>> Projects: http://dbpedia.org, http://nlp2rdf.org,
>> http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
>> <http://www.w3.org/community/ld4lt>
>> Homepage: http://aksw.org/SebastianHellmann
>> Research Group: http://aksw.org
>>
--
All the best,
Sebastian Hellmann
Director of Knowledge Integration and Linked Data Technologies (KILT)
Competence Center
at the Institute for Applied Informatics (InfAI) at Leipzig University
Executive Director of the DBpedia Association
Projects: http://dbpedia.org, http://nlp2rdf.org,
http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
<http://www.w3.org/community/ld4lt>
Homepage: http://aksw.org/SebastianHellmann
Research Group: http://aksw.org
Re: Riot warning for Unicode NFC IRIs
Posted by Laura Morales <la...@mail.com>.
Valid URIs are basically ASCII strings. I don't think you can stick any other character in there.
> Sent: Friday, September 20, 2019 at 9:11 AM
> From: "Sebastian Hellmann" <he...@informatik.uni-leipzig.de>
> To: users@jena.apache.org
> Subject: Riot warning for Unicode NFC IRIs
>
> Hi all,
>
> I really don't understand these RIOT warnings:
>
> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
> <http://dbpedia.org/resource/Ranma_½> Code: 47/NOT_NFKC in PATH: The IRI
> is not in Unicode Normal Form KC.
> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
> <http://dbpedia.org/resource/Ranma_½> Code: 56/COMPATIBILITY_CHARACTER
> in PATH: Bad character
>
> for ½ NFC == NFD as well as NFKD == NFKC
>
> Tested here: https://minaret.info/test/normalize.msp
>
> Result string: ½
> Result in hex: bd
>
> Result string: 1⁄2
> Result in hex: 31 2044 32
>
> Both Wikipedia and DBpedia can not handle NFKC:
> https://en.wikipedia.org/wiki/Ranma_1⁄2
>
> Also the RDF Spec says to normalize to NFC:
> https://www.w3.org/TR/rdf11-concepts/#section-IRIs
>
>
> We are using these versions:
>
> +- org.apache.jena:jena-core:jar:3.7.0:compile
> [INFO] | | \- org.apache.jena:jena-base:jar:3.7.0:compile
>
> Concretely I have these questions:
>
> - can you check whether this is a bug or point me to the issue tracker,
> where this was or should be recorded.
>
> - How can I disable specifically these two warning messages, either by
> configuration or by tuning log4j?
>
>
> --
> All the best,
> Sebastian Hellmann
>
> Director of Knowledge Integration and Linked Data Technologies (KILT)
> Competence Center
> at the Institute for Applied Informatics (InfAI) at Leipzig University
> Executive Director of the DBpedia Association
> Projects: http://dbpedia.org, http://nlp2rdf.org,
> http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
> <http://www.w3.org/community/ld4lt>
> Homepage: http://aksw.org/SebastianHellmann
> Research Group: http://aksw.org
>
Re: Riot warning for Unicode NFC IRIs
Posted by Sebastian Hellmann <he...@informatik.uni-leipzig.de>.
On 20.09.19 16:50, Andy Seaborne wrote:
>> Checking for NFC seems practical as this should be a good goal and it
>> is in the RDF spec and information about it is decent. NOT_NFKC or
>> the Compatibility Character warning is confusing.
>
> Agreed - I'm comfortable with that set of choices.
Ok, thanks, at the moment we have the DBpedia parsing configured to
remove all triples with ERRORs and WARNINGs, however the NOT_NFKC and
Compatibility Warning are pushing out too many triples from the releases
in 140 languages with IRIs. So we will hope for 3.13.0 and this should
be ok for us.
Thanks again.
--
All the best,
Sebastian Hellmann
Director of Knowledge Integration and Linked Data Technologies (KILT)
Competence Center
at the Institute for Applied Informatics (InfAI) at Leipzig University
Executive Director of the DBpedia Association
Projects: http://dbpedia.org, http://nlp2rdf.org,
http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
<http://www.w3.org/community/ld4lt>
Homepage: http://aksw.org/SebastianHellmann
Research Group: http://aksw.org
Re: Riot warning for Unicode NFC IRIs
Posted by Andy Seaborne <an...@apache.org>.
On 20/09/2019 14:37, Sebastian Hellmann wrote:
> Hi Andy,
>
> I would prefer:
>
> // keeping NOT NFC Warnings
>
> // setErrorWarning(iriFactoryInst, ViolationCodes.NOT_NFC, false, false);
>
> // turning of NOT_NFKC warnings
>
> setErrorWarning(iriFactoryInst, ViolationCodes.NOT_NFKC, false, false);
>
> // ** Applies to various unicode blocks.
>
> setErrorWarning(iriFactory, ViolationCodes.COMPATIBILITY_CHARACTER,
> false, false);
>
>
> Reason:
>
> I am really trying to understand Unicode details, but it is tough and I
> am doing this by the side.
Ditto.
jena-iri and its checking is vast and complicated.
Long term, I'd like to switch to a simpler and more maintainable
codebase for the basic use of parsing. But that's an aspiration (An
"aspiration" is a "plan" without a schedule :-)
> Checking for NFC seems practical as this
> should be a good goal and it is in the RDF spec and information about it
> is decent. NOT_NFKC or the Compatibility Character warning is confusing.
Agreed - I'm comfortable with that set of choices.
(the various negations of NOT tests is also confusing!)
>
> I am also wondering, if NOT_NFC and NOT_NFKC would mean that at least
> one warning is always produced. Turning all three off would be acceptable.
>
> Or maybe I am reading this wrong, it first checks whether it is a
> compatibility char and then requires it to be in NFKC for backward
> compatibility?
It's three separate tests. (AbsLexer.java if anyone wants to look)
It's using java.text.Normalizer and some special code for compatibility.
I don't understand Unicode compatibility; the comment is that it is the
difference between NFD and NFKD.
>
> Config fine-tuning:
>
> Regarding changing the config ourselves for further fine - tuning, we
> would copy the class into the current code and adjust it, I assume. Or
> do you think it is possible to make `private static void
> setErrorWarning` public?
When an IRIFactory is first used, it gets frozen and you can't change
the warning/errors anymore. And it get used in the class it is created
in so an app can't change it in-place.
Tinkering with the configuration of IRIResolver to allow it to change
the IRIFactory can be done but we are very near 3.13.0 release so I'm
keen to only make simple change (turning warnings/errors off; these have
been tried out before). Adjusting the IRIFactory risks getting into
weird java initializer situations, altering even something like that.
Something better done earlier in a release cycle.
Andy
>
> -- Sebastian
>
>
>
>
> On 20.09.19 14:48, Andy Seaborne wrote:
>>
>>
>> On 20/09/2019 10:53, Sebastian Hellmann wrote:
>>> On 20.09.19 11:23, Andy Seaborne wrote:
>>
>>>> https://issues.apache.org/jira/browse/JENA-864
>>>>
>>>> Personally, I think we should just turn the warnings off.
>>>
>>>
>>> That would be great. They are harmful in a way that they cause some
>>> flooding in stackoverflow/issue trackers as people are confused
>>> (googled before I wrote).
>>
>> https://github.com/apache/jena/pull/607
>>
>> Andy
>>
Re: Riot warning for Unicode NFC IRIs
Posted by Sebastian Hellmann <he...@informatik.uni-leipzig.de>.
Hi Andy,
I would prefer:
// keeping NOT NFC Warnings
// setErrorWarning(iriFactoryInst, ViolationCodes.NOT_NFC, false, false);
// turning of NOT_NFKC warnings
setErrorWarning(iriFactoryInst, ViolationCodes.NOT_NFKC, false, false);
// ** Applies to various unicode blocks.
setErrorWarning(iriFactory, ViolationCodes.COMPATIBILITY_CHARACTER,
false, false);
Reason:
I am really trying to understand Unicode details, but it is tough and I
am doing this by the side. Checking for NFC seems practical as this
should be a good goal and it is in the RDF spec and information about it
is decent. NOT_NFKC or the Compatibility Character warning is confusing.
I am also wondering, if NOT_NFC and NOT_NFKC would mean that at least
one warning is always produced. Turning all three off would be acceptable.
Or maybe I am reading this wrong, it first checks whether it is a
compatibility char and then requires it to be in NFKC for backward
compatibility?
Config fine-tuning:
Regarding changing the config ourselves for further fine - tuning, we
would copy the class into the current code and adjust it, I assume. Or
do you think it is possible to make `private static void
setErrorWarning` public?
-- Sebastian
On 20.09.19 14:48, Andy Seaborne wrote:
>
>
> On 20/09/2019 10:53, Sebastian Hellmann wrote:
>> On 20.09.19 11:23, Andy Seaborne wrote:
>
>>> https://issues.apache.org/jira/browse/JENA-864
>>>
>>> Personally, I think we should just turn the warnings off.
>>
>>
>> That would be great. They are harmful in a way that they cause some
>> flooding in stackoverflow/issue trackers as people are confused
>> (googled before I wrote).
>
> https://github.com/apache/jena/pull/607
>
> Andy
>
--
All the best,
Sebastian Hellmann
Director of Knowledge Integration and Linked Data Technologies (KILT)
Competence Center
at the Institute for Applied Informatics (InfAI) at Leipzig University
Executive Director of the DBpedia Association
Projects: http://dbpedia.org, http://nlp2rdf.org,
http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
<http://www.w3.org/community/ld4lt>
Homepage: http://aksw.org/SebastianHellmann
Research Group: http://aksw.org
Re: Riot warning for Unicode NFC IRIs
Posted by Andy Seaborne <an...@apache.org>.
On 20/09/2019 10:53, Sebastian Hellmann wrote:
> On 20.09.19 11:23, Andy Seaborne wrote:
>> https://issues.apache.org/jira/browse/JENA-864
>>
>> Personally, I think we should just turn the warnings off.
>
>
> That would be great. They are harmful in a way that they cause some
> flooding in stackoverflow/issue trackers as people are confused (googled
> before I wrote).
https://github.com/apache/jena/pull/607
Andy
Re: Riot warning for Unicode NFC IRIs
Posted by Sebastian Hellmann <he...@informatik.uni-leipzig.de>.
Hi Andy,
On 20.09.19 11:23, Andy Seaborne wrote:
>> Also the RDF Spec says to normalize to NFC:
>> https://www.w3.org/TR/rdf11-concepts/#section-IRIs
>>
>>
>> We are using these versions:
>>
>> +- org.apache.jena:jena-core:jar:3.7.0:compile
>> [INFO] | | \- org.apache.jena:jena-base:jar:3.7.0:compile
>
> Update!
>
Aye sir, scheduled.
>>
>> Concretely I have these questions:
>>
>> - can you check whether this is a bug or point me to the issue
>> tracker, where this was or should be recorded.
>
> https://issues.apache.org/jira/browse/JENA-864
>
> Personally, I think we should just turn the warnings off.
That would be great. They are harmful in a way that they cause some
flooding in stackoverflow/issue trackers as people are confused (googled
before I wrote).
I took your advice from CDL last year and we developed extra tests that
not only benchmark URI/IRI validity, but actually same strings. It is
still rudimentary, but it catches non-sameness of DBpedia URIs across
releases, i.e. having an `?` in URIs is valid but should never occur in
DBpedia URIs who never ever have a query path: [1]
> Code: org.apache.jena.riot.system.IRIResolver although IIRC the IRI
> factor setup is frozen and can't be changed later in app code but
> that's from my memory only.
Yes, looks like "private final2 at least in 3.7. These are quite useful
settings in the code comments, maybe this could be an insertion point
for us for tests in [1]
Thanks for the pointers, very helpful,
Sebastian
[1]
https://github.com/dbpedia/extraction-framework/blob/master/dump/src/test/resources/dbpedia-specific-ci-tests.ttl
--
All the best,
Sebastian Hellmann
Director of Knowledge Integration and Linked Data Technologies (KILT)
Competence Center
at the Institute for Applied Informatics (InfAI) at Leipzig University
Executive Director of the DBpedia Association
Projects: http://dbpedia.org, http://nlp2rdf.org,
http://linguistics.okfn.org, https://www.w3.org/community/ld4lt
<http://www.w3.org/community/ld4lt>
Homepage: http://aksw.org/SebastianHellmann
Research Group: http://aksw.org
Re: Riot warning for Unicode NFC IRIs
Posted by Andy Seaborne <an...@apache.org>.
On 20/09/2019 08:11, Sebastian Hellmann wrote:
> Hi all,
>
> I really don't understand these RIOT warnings:
>
> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
> <http://dbpedia.org/resource/Ranma_½> Code: 47/NOT_NFKC in PATH: The IRI
> is not in Unicode Normal Form KC.
> 19/09/20 08:43:00 WARN riot: [line: 229, col: 1 ] Bad IRI:
> <http://dbpedia.org/resource/Ranma_½> Code: 56/COMPATIBILITY_CHARACTER
> in PATH: Bad character
>
> for ½ NFC == NFD as well as NFKD == NFKC
>
> Tested here: https://minaret.info/test/normalize.msp
>
> Result string: ½
> Result in hex: bd
Then the UTF-8 encoding as C2 BD.
>
> Result string: 1⁄2
> Result in hex: 31 2044 32
> Both Wikipedia and DBpedia can not handle NFKC:
> https://en.wikipedia.org/wiki/Ranma_1⁄2
Different issue. Technically IRIs are sort of "virtual" and mapped to
URIs. RFC 3987 says:
A mapping from IRIs to
URIs is defined, which means that IRIs can be used instead of URIs,
where appropriate, to identify resources.
Wikipedia and DBpedia are working in URIs.
They have to deal with the fact the "same thing" can be written
different ways. They choose to use the URI form: I guess because they
emphasis HTTP usage.
Browsers hide all this from the users and will convert to URI form from
the browser bar.
(Jena works internally in IRIs.)
>
> Also the RDF Spec says to normalize to NFC:
> https://www.w3.org/TR/rdf11-concepts/#section-IRIs
>
>
> We are using these versions:
>
> +- org.apache.jena:jena-core:jar:3.7.0:compile
> [INFO] | | \- org.apache.jena:jena-base:jar:3.7.0:compile
Update!
>
> Concretely I have these questions:
>
> - can you check whether this is a bug or point me to the issue tracker,
> where this was or should be recorded.
https://issues.apache.org/jira/browse/JENA-864
Personally, I think we should just turn the warnings off.
>
> - How can I disable specifically these two warning messages, either by
> configuration or by tuning log4j?
Code: org.apache.jena.riot.system.IRIResolver although IIRC the IRI
factor setup is frozen and can't be changed later in app code but that's
from my memory only.
The logger is "org.apache.jena.riot"
The command line tools look for a local log4j.properties file in
addition to the usual log4j initialization mechanism, then fall back to
a built-in setup.
Andy
Re: Riot warning for Unicode NFC IRIs
Posted by Laura Morales <la...@mail.com>.
> Both Wikipedia and DBpedia can not handle NFKC:
> https://en.wikipedia.org/wiki/Ranma_1⁄2
The wikipedia link is https://en.wikipedia.org/wiki/Ranma_%C2%BD